Optimization

Adam Optimizer

Adam is an adaptive optimizer that combines momentum (EMA of gradients) with per-parameter RMS normalization (EMA of squared gradients).

status: publishedimportance: criticaldifficulty 3/5math: undergraduateread: 16mlive demo

Back to Optimization Next: Weight Decay & AdamW: Decoupled Regularization

Editorial optimization illustration of adaptive Adam steps using momentum and second-moment scaling across a loss surface.

Concept Structure

Adam Optimizer

01Intuition

Start with the picture, metaphor, or geometric mechanism.

02Math

Make the objects explicit and connect them with notation.

03Code

Mirror the equations with runnable implementation details.

04Interactive Demo

Manipulate the mechanism and watch the idea respond.

2prerequisites

2next concepts

2related links

Learning map

Adam Optimizer

BeforeDerivativesNow4/4 sections readyTryManipulate one control and predict the visible change.NextWeight Decay & AdamW: Decoupled Regularization

Object flow

4/4 sections readyAsk about this Research room

ConceptAdam OptimizerOptimization EquationAdam Optimizer equation 1Exact equation object CodeAdam Optimizer code witness 1Exact code witness DemoAdam Optimizer interactive demoVisualization object ClaimAdam maintains exponential moving averages of gradients and squared g...Exact claim check SourceAdam: A Method for Stochastic OptimizationExact source object

ConceptAdam OptimizerOptimization

1 source attachedLocal snapshot ready

concept:optimization/adam

Codewitness nearby Predictbefore reveal Roomobject handoff

Conceptual Bridge

What should feel connected as you move through this page.

Carry inDerivatives

Bring the mental model from Derivatives; this page will reuse it instead of restarting from zero.

Work hereAdam Optimizer

Adam is an adaptive optimizer that combines momentum (EMA of gradients) with per-parameter RMS normalization (EMA of squared gradients).

Carry outWeight Decay & AdamW: Decoupled Regularization

The next edge should feel earned: use the demo prediction here before following Weight Decay & AdamW: Decoupled Regularization.

Test the linkManipulate one control and predict the visible change.Then continue to Weight Decay & AdamW: Decoupled Regularization

01IntuitionStart with the picture, metaphor, or geometric mechanism.02MathMake the objects explicit and connect them with notation.03CodeMirror the equations with runnable implementation details.04Interactive DemoManipulate the mechanism and watch the idea respond.

Intuition

Build the mental picture first so the rest of the page has something to attach to.

Section prompt

Training is noisy: mini-batch gradients bounce around, and different parameters can have very different natural scales.

Adam combines two simple stabilizers:

Momentum: smooth the gradient over time, so you don’t overreact to one noisy batch.
RMS normalization: keep an exponential moving average of squared gradients so each parameter gets a step size that matches its typical gradient scale.

A useful mental model is: Adam maintains a per-parameter “velocity” (direction) and a per-parameter “uncertainty/scale” (how big gradients usually are), then divides one by the other.

Math

Translate the story into symbols, assumptions, and a derivation you can inspect.

Section prompt

Equation 1m_t = \beta_1 m_{t-1} + (1-\beta_1) g_t Equation 2v_t = \beta_2 v_{t-1} + (1-\beta_2) g_t^2

Let $g_t = \nabla_\theta \mathcal{L}_t(\theta_t)$ be the (stochastic) gradient at step $t$ .

Adam keeps exponential moving averages:

m_t = \beta_1 m_{t-1} + (1-\beta_1) g_t

v_t = \beta_2 v_{t-1} + (1-\beta_2) g_t^2

Because $m_t$ and $v_t$ start at zero, Adam uses bias correction:

\hat m_t = \frac{m_t}{1-\beta_1^t}, \qquad \hat v_t = \frac{v_t}{1-\beta_2^t}

Update rule:

\theta_{t+1} = \theta_t - \alpha \frac{\hat m_t}{\sqrt{\hat v_t} + \varepsilon}.

The division is elementwise, so Adam acts like a diagonal preconditioner: coordinates with consistently large gradients get smaller effective steps, while coordinates with consistently small gradients get larger relative steps. Bias correction matters most early in training because the moving averages are initialized at zero. The small $\varepsilon$ is not a learning signal; it prevents division by zero and can affect stability when gradients are tiny.

Typical defaults: $\beta_1=0.9$ , $\beta_2=0.999$ , $\varepsilon=10^{-8}$ .

Code

Keep the implementation aligned with the notation so the algorithm is legible.

Section prompt

Code witness 1import torch alpha = 0.1 beta1, beta2 = 0.9, 0.999 eps = 1e-8 theta = torch.tensor([3.0], req...python

import torch

alpha = 0.1
beta1, beta2 = 0.9, 0.999
eps = 1e-8

theta = torch.tensor([3.0], requires_grad=True)
m = torch.zeros_like(theta)
v = torch.zeros_like(theta)

for t in range(1, 51):
    loss = (theta ** 2).sum()
    loss.backward()

    g = theta.grad.detach()
    m = beta1 * m + (1 - beta1) * g
    v = beta2 * v + (1 - beta2) * (g * g)

    mhat = m / (1 - beta1 ** t)
    vhat = v / (1 - beta2 ** t)

    theta = (theta - alpha * mhat / (vhat.sqrt() + eps)).detach().requires_grad_(True)

    if t in [1, 5, 10, 50]:
        print(t, "theta=", theta.item(), "loss=", loss.item())

Interactive Demo

Use direct manipulation to connect the explanation to a moving system.

Section prompt

Use the demo to compare Adam to SGD variants and see how the moving averages change the effective step size.

Live Concept Demo

Explore Adam Optimizer

The stage is code-native and interactive. Use it to test the explanation against the mechanism.

difficulty 3/5undergraduatecode-aligned

Demo Prediction Checkpoint

Manipulate one control and predict the visible change.

Commit to what Adam Optimizer should make visible before reading the result.

After The First Pass

Turn the concept into an inspected object.

Once the invariant is visible in the intuition, math, code, and demo, use these panels to inspect the mechanism visually, check source support, practice the idea, and attach a grounded research question.

Mechanism Storyboard

See the idea move before the page explains it

Adam is an adaptive optimizer that combines momentum (EMA of gradients) with per-parameter RMS normalization (EMA of squared gradients).

Prediction open01 / Intuition

Prediction lens

Start with the picture, metaphor, or geometric mechanism.

Commit first

Before reading further, choose the kind of change Adam Optimizer should make visible.

Visual Inquiry

Make the image answer a mathematical question

Adam is an adaptive optimizer that combines momentum (EMA of gradients) with per-parameter RMS normalization (EMA of squared gradients).

4/4 stages readyLive demo connected

Visual cueWhich visible object should carry the first intuition?

Inspection depth2/4

Prediction

Which visible object should carry the first intuition?

Commit first

Pick the cue that should make Adam Optimizer easier to reason about before the page gives the answer.

Source Grounding

Canonical references for the mechanism on this page.

paper · 2014Adam: A Method for Stochastic OptimizationKingma and Ba

Grounds Adam's first- and second-moment estimates, bias correction, and adaptive step-size mechanics.

Open source

Claim Review

Adam is an adaptive optimizer that combines momentum (EMA of gradients) with per-parameter RMS normalization (EMA of squared gradients).

Status1 substantive review recorded

Claims without a substantive review badge still need exact source-support review.

Sources1 reference

kingma-2014-adam

Witnesses4 local objects

Use equation, code, and demo objects to check whether the source support is operational.

Substantively reviewedAdam maintains exponential moving averages of gradients and squared gradients, bias-corrects both from zero initialization, and takes elementwise adaptive steps by subtracting alpha*mhat_t/(sqrt(vhat_t)+epsilon).Claim metadata: source checked

Kingma and Ba's Algorithm 1 initializes m0 and v0 to zero, updates biased first-moment and second raw-moment estimates from gt and gt^2, computes bias-corrected estimates, and updates parameters with alpha*mhat_t/(sqrt(vhat_t)+epsilon), with vector operations elementwise. The page's equations, code witness, and demo instantiate those mechanics and expose the per-coordinate effective step scaling.

Sources: Adam: A Method for Stochastic OptimizationThis checks Adam's original adaptive-moment update mechanics, not convergence guarantees, AdamW decoupled weight decay, AMSGrad, optimizer generalization debates, sparse-gradient variants, or framework-specific implementation details.A bounded review summary is present; still check caveats and exact source scope.

Checked Kingma and Ba Algorithm 1 plus sections 2 and 3: the paper initializes m0 and v0 as zero vectors, updates biased first and second raw moment estimates from gt and gt^2, divides by 1-beta1^t and 1-beta2^t for bias correction, states vector operations are element-wise, and updates theta with alpha*mhat/(sqrt(vhat)+epsilon). Scope is original Adam update mechanics only.

Reviewer: codex+oracle; reviewed 2026-05-06

source-span-kingma-2014-adam math-object-1 math-object-2 code-witness-1 interactive-demo

Source support candidates

paper 2014Adam: A Method for Stochastic Optimization

Grounds Adam's first- and second-moment estimates, bias correction, and adaptive step-size mechanics.

Mechanism witnesses

Equation 1

m_t = \beta_1 m_{t-1} + (1-\beta_1) g_t

Equation 2

v_t = \beta_2 v_{t-1} + (1-\beta_2) g_t^2

Code witness 1import torch alpha = 0.1 beta1, beta2 = 0.9, 0.999 eps = 1e-8 theta = torch.tensor([3.0], req...Demo stateLive mechanism probe

Practice Loop

Try the idea before it explains itself

Adam is an adaptive optimizer that combines momentum (EMA of gradients) with per-parameter RMS normalization (EMA of squared gradients).

Readiness0/3 checks ready

Predict

Before touching the demo, predict one visible change that should happen in Adam Optimizer.

Hint 1

Reveal when your model needs a nudge.

Hint 2

Reveal when your model needs a nudge.

Hint 3

Reveal when your model needs a nudge.

Your answer canvas

Local checks

Claim

A concrete answer is on the canvas.

Mechanism

The answer names why the claim should hold.

Bridge

It touches the page context or a neighboring idea.

Misconception check

Object research drawerClose

ConceptAdam OptimizerOptimization

Code witness comparisonAdam Optimizer code witness 1alpha = 0.1 Prediction before revealAdam Optimizer interactive demoManipulate one control and predict the visible change.

Grounded room questionWhat is the smallest example that makes Adam Optimizer click without losing the math?Local snapshot ready

Research Room

Attach the question to an exact object

Pick the concept, equation, source, code witness, claim, misconception, or demo state before asking for help. The handoff stays grounded to that object.

Next local actionNo local draft saved yet

Open the draft below to save one note and next action in this browser.

conceptOptimization

Adam Optimizer

Anchored question

What is the smallest example that makes Adam Optimizer click without losing the math?

Local action draftNo local draft saved yetExpand only when ready to capture one local next action

Local action draft

This draft stays locally in this browser for concept:optimization/adam.

Draft noteNext action

No local draft saved.

Evidence to inspect

Source ids to inspect: kingma-2014-adam
Definition, prerequisite, and contrast concept links
The equation or code witness that makes the concept operational
One demo state that shows the invariant instead of a slogan

What would resolve this

The learner can state the mechanism in their own words
The learner can name the prerequisite that would repair confusion
The learner can predict how the mechanism changes under one perturbation

Grounded AI handoff

I am working in Continuous Function's research reading room. Object: concept - Adam Optimizer Object key: concept:optimization/adam Context: Optimization Anchor id: concept/concept-notebook/optimization/adam Open question: What is the smallest example that makes Adam Optimizer click without losing the math? Evidence to inspect: - Source ids to inspect: kingma-2014-adam - Definition, prerequisite, and contrast concept links - The equation or code witness that makes the concept operational - One demo state that shows the invariant instead of a slogan What would resolve this: - The learner can state the mechanism in their own words - The learner can name the prerequisite that would repair confusion - The learner can predict how the mechanism changes under one perturbation Answer as a careful research tutor: stay source-grounded, separate verified evidence from assumptions, name the relevant math objects, and end with one next action.

Open source object

concept/concept-notebook/optimization/adam
concept:optimization/adam

Learning Map

Before / Now / Try / Next

BeforeDerivatives

NowIntuition → Math → Code → Demo

TryManipulate one control and predict the visible change.

NextWeight Decay & AdamW: Decoupled Regularization

Intuitionready
Mathready
Codeready
Interactive Demoready

Object Companion

Ask beside the selected object

Adam is an adaptive optimizer that combines momentum (EMA of gradients) with per-parameter RMS normalization (EMA of squared gradients).

Your question

GoalComfortStyleStuck on

Context prompt

You are my AI learning companion for Continuous Function. Current context: Optimization concept. Learning surface: Adam Optimizer. What this page says: Adam is an adaptive optimizer that combines momentum (EMA of gradients) with per-parameter RMS normalization (EMA of squared gradients). Current section: Intuition, math, code, and interactive demo. Suggested next step: Manipulate one control and predict the visible change.. Learner goal: Understand the idea. Learner comfort level: New to this. Preferred explanation style: Visual first. Task: Explain the central idea in plain language, then restate it with the exact math objects from the page. Answer in a way that helps me learn: ask one clarifying question only if needed, use intuition before notation, and end with one thing I should try on the page.

Domain

Optimization

optimizationdeep-learningadam

Prerequisites

Derivatives Norms

Leads To

Weight Decay & AdamW: Decoupled Regularization Learning Rate Schedules: Warmup, Decay & Cycling

Loss Landscapes, Sharpness & Flat Minima Gradient Clipping & Explosion Prevention

Within this domain