Continuous Function

Machine Learning

BatchNorm, LayerNorm, and RMSNorm

Normalization methods differ by axis: BatchNorm couples examples through batch statistics, while LayerNorm and RMSNorm normalize one example/token vector at a time.

status: reviewimportance: criticaldifficulty 3/5math: graduateread: 18mlive demo
Back to Machine LearningNext: Layer Normalization & RMSNorm

Concept Structure

BatchNorm, LayerNorm, and RMSNorm

01Intuition

Start with the picture, metaphor, or geometric mechanism.

02Math

Make the objects explicit and connect them with notation.

03Code

Mirror the equations with runnable implementation details.

04Interactive Demo

Manipulate the mechanism and watch the idea respond.

1prerequisites
3next concepts
3related links

Learner Contract

What this page should let you do.

You are here becauseNormalization methods differ by axis: BatchNorm couples examples through batch statistics, while LayerNorm and RMSNorm normalize one example/token vector at a time.

This Machine Learning concept is the current idea: keep the same invariant visible across Intuition, Math, Code, Interactive Demo.

By the end4/4 sections ready | runnable code expected | live demo

Explain the mechanism, trace the main notation, and test one prediction in the live demo.

Do this firstIntuition

Read the intuition before the notation; the math should name a mechanism you already felt.

Then go nextLayer Normalization & RMSNorm

Follow this edge after making one prediction here; the next page should reuse the result, not restart the route.

Test the linkManipulate one control and predict the visible change.Then continue to Layer Normalization & RMSNorm

Claim/source review status

Substantive review recorded

1/1 claims have bounded review metadata; still check caveats and source scope.Metadata-derived; review may be AI-assisted. Not a human certification.
Claims1/1 reviewed
Sources5 cited
Codeattached
Demolive
Reviewed2026-07-02
Updatedpage 2026-07-02

Learning item flow

4/4 sections readyAsk about thisResearch room
ConceptBatchNorm, LayerNorm, and RMSNormMachine LearningEquationBatchNorm, LayerNorm, and RMSNorm equation 1Exact equationCodeBatchNorm, LayerNorm, and RMSNorm runnable code 1Exact runnable codeDemoBatchNorm, LayerNorm, and RMSNorm interactive demoInteractive demoClaimBatchNorm couples examples through per-channel minibatch statistics;...Exact claim checkSourceDive into Deep Learning: Batch NormalizationExact source
ConceptBatchNorm, LayerNorm, and RMSNormMachine Learning
5 sources attachedLocal snapshot ready
concept:machine-learning/normalization-batch-layer-rms
CodenearbyPredictbefore revealRoomitem handoff
01IntuitionStart with the picture, metaphor, or geometric mechanism.02MathMake the objects explicit and connect them with notation.03CodeMirror the equations with runnable implementation details.04Interactive DemoManipulate the mechanism and watch the idea respond.
01

01

Intuition

Build the mental picture first so the rest of the page has something to attach to.

Section prompt

You are here because normalization is often taught as a bag of formulas, when the first question should be simpler: which axes are being coupled?

Before this, know activation functions and why scale affects gradient flow. By the end, you should be able to look at a tensor and say which values contribute to the mean, variance, or RMS used to normalize one highlighted activation.

BatchNorm looks down the batch for one channel. If you change another example in the same mini-batch, the normalized value for the target example can change because the batch mean and variance changed. That coupling is useful in many settings, especially CNN-style training, but it also creates batch-size sensitivity and a training-vs-inference distinction.

LayerNorm looks across the features of one example or token. If you swap another example in the batch, the target token's LayerNorm output does not change. The statistics came from the target vector itself.

RMSNorm keeps the same per-example/per-token frame but removes mean-centering. It divides by the root mean square of that vector. So RMSNorm shares LayerNorm's batch independence, but it preserves any mean offset that LayerNorm would subtract.

The core distinction:

  • BatchNorm: "Who else is in my mini-batch?"
  • LayerNorm: "What is the mean and spread of this vector?"
  • RMSNorm: "What is the RMS scale of this vector?"

This page is not ranking the methods. It is teaching the axis choice that makes later transformer, CNN, and training-stability discussions less slippery.

02

02

Math

Translate the story into symbols, assumptions, and a derivation you can inspect.

Section prompt
Equation 1μjBN=1B∑b=1BXb,j,(σjBN)2=1B∑b=1B(Xb,j−μjBN)2.\mu^{\mathrm{BN}}_j=\frac{1}{B}\sum_{b=1}^{B}X_{b,j}, \qquad (\sigma^{\mathrm{BN}}_j)^2=\frac{1}{B}\sum_{b=1}^{B}(X_{b,j}-\mu^{\mathrm{BN}}_j)^2.Equation 2BN(X)b,j=γjXb,j−μjBN(σjBN)2+ϵ+βj.\mathrm{BN}(X)_{b,j} = \gamma_j\frac{X_{b,j}-\mu^{\mathrm{BN}}_j}{\sqrt{(\sigma^{\mathrm{BN}}_j)^2+\epsilon}} +\beta_j.

Let X∈RB×DX\in\mathbb R^{B\times D} be a batch of BB examples and DD channels/features. We will focus on the target value Xb,jX_{b,j}.

For a fully connected activation matrix, BatchNorm normalizes channel jj using batch statistics:

μjBN=1B∑b=1BXb,j,(σjBN)2=1B∑b=1B(Xb,j−μjBN)2.\mu^{\mathrm{BN}}_j=\frac{1}{B}\sum_{b=1}^{B}X_{b,j}, \qquad (\sigma^{\mathrm{BN}}_j)^2=\frac{1}{B}\sum_{b=1}^{B}(X_{b,j}-\mu^{\mathrm{BN}}_j)^2.

Then

BN(X)b,j=γjXb,j−μjBN(σjBN)2+ϵ+βj.\mathrm{BN}(X)_{b,j} = \gamma_j\frac{X_{b,j}-\mu^{\mathrm{BN}}_j}{\sqrt{(\sigma^{\mathrm{BN}}_j)^2+\epsilon}} +\beta_j.

The statistics for Xb,jX_{b,j} use other examples Xb′,jX_{b',j} in the same mini-batch. During inference, BatchNorm usually uses running estimates rather than the current batch.

LayerNorm normalizes one example vector x=Xb,:∈RDx=X_{b,:}\in\mathbb R^D across its features:

μLN(x)=1D∑j=1Dxj,(σLN(x))2=1D∑j=1D(xj−μLN(x))2.\mu^{\mathrm{LN}}(x)=\frac{1}{D}\sum_{j=1}^{D}x_j, \qquad (\sigma^{\mathrm{LN}}(x))^2=\frac{1}{D}\sum_{j=1}^{D}(x_j-\mu^{\mathrm{LN}}(x))^2.

Then

LN(x)j=γjxj−μLN(x)(σLN(x))2+ϵ+βj.\mathrm{LN}(x)_j = \gamma_j\frac{x_j-\mu^{\mathrm{LN}}(x)}{\sqrt{(\sigma^{\mathrm{LN}}(x))^2+\epsilon}} +\beta_j.

RMSNorm keeps the per-vector axis but drops mean-centering:

RMSϵ(x)=1D∑j=1Dxj2+ϵ,RMSNorm(x)j=γjxjRMSϵ(x).\mathrm{RMS}_{\epsilon}(x) = \sqrt{\frac{1}{D}\sum_{j=1}^{D}x_j^2+\epsilon}, \qquad \mathrm{RMSNorm}(x)_j = \gamma_j\frac{x_j}{\mathrm{RMS}_{\epsilon}(x)}.

The small but important consequence: if we replace some other row Xb′,:X_{b',:} while keeping target row Xb,:X_{b,:} fixed, BatchNorm can change the target output because μjBN\mu^{\mathrm{BN}}_j and σjBN\sigma^{\mathrm{BN}}_j changed. LayerNorm and RMSNorm do not change for the target row because their statistics only use Xb,:X_{b,:}.

03

03

Code

Keep the implementation aligned with the notation so the algorithm is legible.

Section prompt
Runnable code 1import numpy as np X = np.array([ [ 0.6, -1.2, 2.0, 0.3], [-0.4, 0.7, -1.1, 1.0], [ 1.5, -0.3...python
import numpy as np

X = np.array([
    [ 0.6, -1.2,  2.0,  0.3],
    [-0.4,  0.7, -1.1,  1.0],
    [ 1.5, -0.3,  0.2, -0.7],
    [-2.0,  1.3,  0.5, -1.5],
])
X_swap = X.copy()
X_swap[3] = np.array([2.4, -1.6, -3.0, 0.4])
eps = 1e-5

def batchnorm(A):
    mu = A.mean(axis=0, keepdims=True)
    var = ((A - mu) ** 2).mean(axis=0, keepdims=True)
    return (A - mu) / np.sqrt(var + eps)

def layernorm(A):
    mu = A.mean(axis=1, keepdims=True)
    var = ((A - mu) ** 2).mean(axis=1, keepdims=True)
    return (A - mu) / np.sqrt(var + eps)

def rmsnorm(A):
    rms = np.sqrt((A ** 2).mean(axis=1, keepdims=True) + eps)
    return A / rms

target = (0, 2)
for name, fn in [("BN", batchnorm), ("LN", layernorm), ("RMS", rmsnorm)]:
    before = fn(X)[target]
    after = fn(X_swap)[target]
    print(name, round(before, 3), round(after, 3), "changed:", abs(after - before) > 1e-6)

The runnable code checks the axis claim. Only the last example changes, yet the target BatchNorm value changes because the channel statistics changed. LayerNorm and RMSNorm stay fixed for the target row.

04

04

Interactive Demo

Use direct manipulation to connect the explanation to a moving system.

Section prompt

Use the Normalization Axis Lab to inspect a highlighted value in a small matrix. Before reveal, predict what happens when a different example in the same batch is swapped:

  • Only BatchNorm changes: batch statistics for the highlighted channel changed.
  • LayerNorm changes too: the target row's feature statistics changed.
  • RMSNorm changes too: the target row's RMS changed.
  • None change: the swapped example is irrelevant to all three methods.

The correct lesson is that BatchNorm is batch-coupled, while LayerNorm and RMSNorm are per-example/per-token for this finite matrix witness. After reveal, compare the axis cards, the target output before/after swap, and the caveat strip. The lab is not an architecture benchmark; it is a mechanism check for which values each normalization method lets into the statistic.

Live Concept Demo

Explore BatchNorm, LayerNorm, and RMSNorm

The stage is code-native and interactive. Use it to test the explanation against the mechanism.

difficulty 3/5graduatecode-aligned
Demo Prediction Checkpoint

Manipulate one control and predict the visible change.

Commit to what BatchNorm, LayerNorm, and RMSNorm should make visible before reading the result.

After The First Pass

Turn the concept into an inspected object.

Once the invariant is visible in the intuition, math, code, and demo, use these panels to inspect the mechanism visually, check source support, practice the idea, and attach a grounded research question.

Mechanism Storyboard

See the idea move before the page explains it

Normalization methods differ by axis: BatchNorm couples examples through batch statistics, while LayerNorm and RMSNorm normalize one example/token vector at a time.

Prediction open01 / Intuition
Prediction lens

Start with the picture, metaphor, or geometric mechanism.

Commit first

Before reading further, choose the kind of change BatchNorm, LayerNorm, and RMSNorm should make visible.

Visual Inquiry

Make the image answer a mathematical question

Normalization methods differ by axis: BatchNorm couples examples through batch statistics, while LayerNorm and RMSNorm normalize one example/token vector at a time.

4/4 stages readyLive demo connected
Prediction

Which visible object should carry the first intuition?

Commit first

Pick the cue that should make BatchNorm, LayerNorm, and RMSNorm easier to reason about before the page gives the answer.

Source Grounding

Canonical references for the mechanism on this page.

book · 2026Dive into Deep Learning: Batch NormalizationZhang, Lipton, Li, and Smola

Supports minibatch statistics, learned scale/shift, batch-size sensitivity, training/prediction behavior, and practical caveats.

Open source
book · 2026Dive into Deep Learning: Transformer Architecture, Residual Connection and Layer NormalizationZhang, Lipton, Li, and Smola

Directly contrasts BatchNorm and LayerNorm dimensions with a tiny matrix witness.

Open source
paper · 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate ShiftIoffe and Szegedy

Primary BatchNorm source: normalization is part of the architecture and uses each training mini-batch.

Open source
paper · 2016Layer NormalizationBa, Kiros, and Hinton

Primary LayerNorm source: computes mean/variance over all summed inputs in one layer on a single training case.

Open source
paper · 2019Root Mean Square Layer NormalizationZhang and Sennrich

Primary RMSNorm source: removes mean-centering and rescales by root mean square.

Open source

Claim Review

Normalization methods differ by axis: BatchNorm couples examples through batch statistics, while LayerNorm and RMSNorm normalize one example/token vector at a time.

Status1 substantive review recorded

Claims without a substantive review badge still need exact source-support review.

Sources5 references

d2l-2026-batch-normalization, d2l-2026-transformer-layer-vs-batch, ioffe-2015-batch-normalization, ba-2016-layer-normalization, zhang-2019-rmsnorm

Local checks4 local checks

Use equations, runnable code, and demos to check whether the source support is operational.

Substantively reviewedBatchNorm couples examples through per-channel minibatch statistics; LayerNorm uses one example/token's feature mean and variance; RMSNorm uses one example/token's RMS without mean-centering.Claim metadata: source checked

D2L and Ioffe/Szegedy support BatchNorm minibatch statistics; D2L and Ba et al. support LayerNorm feature statistics within a single example; Zhang/Sennrich support RMSNorm as RMS-based rescaling without mean-centering.

Sources: Dive into Deep Learning: Batch Normalization, Dive into Deep Learning: Transformer Architecture, Residual Connection and Layer Normalization, Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift, Layer Normalization, Root Mean Square Layer NormalizationFinite matrix axis witness only; not a claim about norm placement, CNN/Transformer accuracy, distributed sync costs, optimizer choice, inference latency, or universal training stability.A bounded review summary is present; still check caveats and exact reference scope.

Checked D2L BatchNorm, D2L Transformer LayerNorm comparison, Ioffe/Szegedy, Ba/Kiros/Hinton, and Zhang/Sennrich. Sources support minibatch coupling for BatchNorm, per-case feature statistics for LayerNorm, and RMS-only per-vector rescaling for RMSNorm. The demo is a finite-matrix axis witness, not a transformer architecture benchmark or stability guarantee. GPT Pro/Oracle publication critique remains pending.

Reviewer: codex-local-primary-source-audit; reviewed 2026-07-02
source-span-d2l-2026-batch-normalizationsource-span-d2l-2026-transformer-layer-vs-batchsource-span-ba-2016-layer-normalizationsource-span-zhang-2019-rmsnormEquation 1Equation 2Runnable code 1Interactive demo

Source support candidates

book 2026Dive into Deep Learning: Batch Normalization

Supports minibatch statistics, learned scale/shift, batch-size sensitivity, training/prediction behavior, and practical caveats.

book 2026Dive into Deep Learning: Transformer Architecture, Residual Connection and Layer Normalization

Directly contrasts BatchNorm and LayerNorm dimensions with a tiny matrix witness.

paper 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Primary BatchNorm source: normalization is part of the architecture and uses each training mini-batch.

paper 2016Layer Normalization

Primary LayerNorm source: computes mean/variance over all summed inputs in one layer on a single training case.

Mechanism checks

Equation 1
μjBN=1B∑b=1BXb,j,(σjBN)2=1B∑b=1B(Xb,j−μjBN)2.\mu^{\mathrm{BN}}_j=\frac{1}{B}\sum_{b=1}^{B}X_{b,j}, \qquad (\sigma^{\mathrm{BN}}_j)^2=\frac{1}{B}\sum_{b=1}^{B}(X_{b,j}-\mu^{\mathrm{BN}}_j)^2.
Equation 2
BN(X)b,j=γjXb,j−μjBN(σjBN)2+ϵ+βj.\mathrm{BN}(X)_{b,j} = \gamma_j\frac{X_{b,j}-\mu^{\mathrm{BN}}_j}{\sqrt{(\sigma^{\mathrm{BN}}_j)^2+\epsilon}} +\beta_j.
Runnable code 1import numpy as np X = np.array([ [ 0.6, -1.2, 2.0, 0.3], [-0.4, 0.7, -1.1, 1.0], [ 1.5, -0.3...Demo stateLive mechanism probe

Practice Loop

Try the idea before it explains itself

Normalization methods differ by axis: BatchNorm couples examples through batch statistics, while LayerNorm and RMSNorm normalize one example/token vector at a time.

Readiness0/3 checks ready
Predict

Before touching the demo, predict one visible change that should happen in BatchNorm, LayerNorm, and RMSNorm.

Hint 1

Reveal when your model needs a nudge.

Hint 2

Reveal when your model needs a nudge.

Hint 3

Reveal when your model needs a nudge.

Grounded research drawerClose
ConceptBatchNorm, LayerNorm, and RMSNormMachine Learning
Runnable code comparisonBatchNorm, LayerNorm, and RMSNorm runnable code 1X = np.array([Prediction before revealBatchNorm, LayerNorm, and RMSNorm interactive demoManipulate one control and predict the visible change.
Grounded room questionWhat is the smallest example that makes BatchNorm, LayerNorm, and RMSNorm click without losing the math?Local snapshot ready

Research Room

Attach the question to a claim, equation, code, or demo

Pick the concept, equation, source, runnable code, claim, misconception, or demo state before asking for help. The handoff keeps that page item in context.
Next local actionNo local draft saved yet

Open the draft below to save one note and next action in this browser.

conceptMachine Learning

BatchNorm, LayerNorm, and RMSNorm

Attached question

What is the smallest example that makes BatchNorm, LayerNorm, and RMSNorm click without losing the math?

Local action draftNo local draft saved yetExpand only when ready to capture one local next action
Local action draft

This draft stays in this browser, attached to the selected learning item.

No local draft saved.
Evidence to inspect
  • References to inspect: attached references on this page.
  • Definition, prerequisite, and contrast concept links
  • The equation or runnable code that makes the concept operational
  • One demo state that shows the invariant instead of a slogan
What would resolve this
  • The learner can state the mechanism in their own words
  • The learner can name the prerequisite that would repair confusion
  • The learner can predict how the mechanism changes under one perturbation
Grounded AI handoff

I am working in Continuous Function's research reading room. Object: concept - BatchNorm, LayerNorm, and RMSNorm Selected item key: recorded for copy. Context: Machine Learning Page anchor: recorded for copy. Open question: What is the smallest example that makes BatchNorm, LayerNorm, and RMSNorm click without losing the math? Evidence to inspect: - References to inspect: attached references on this page. - Definition, prerequisite, and contrast concept links - The equation or runnable code that makes the concept operational - One demo state that shows the invariant instead of a slogan What would resolve this: - The learner can state the mechanism in their own words - The learner can name the prerequisite that would repair confusion - The learner can predict how the mechanism changes under one perturbation Answer as a careful research tutor: stay source-grounded, separate verified evidence from assumptions, name the relevant math objects, and end with one next action.

View it in context
concept/concept-notebook/machine-learning/normalization-batch-layer-rms concept:machine-learning/normalization-batch-layer-rms