Adam Optimizer

Adam combines momentum, a running average of gradients, with a per-parameter step size scaled by a running average of squared gradients.

Intermediate · Undergraduate mathematics · about 16 minutes

Reading map and next steps

Intuition

The idea in plain words, before the symbols.

Training is noisy: mini-batch gradients bounce around from step to step, and different parameters can have gradients of very different sizes. A single learning rate can then be too large for some parameters and too small for others.

Adam (Kingma and Ba, 2015) keeps two running averages for every parameter:

$m$ , an average of recent gradients. This is momentum: it smooths out noise from individual mini-batches and keeps the direction that recent gradients agree on.
$v$ , an average of recent squared gradients. This estimates the gradient's second raw moment $\mathbb E[g^2]$ , which Kingma and Ba also call the uncentered variance. Because $\mathbb E[g^2]=(\mathbb E g)^2+\operatorname{Var}g$ , it measures how large the gradient typically is, mean and spread together; it is not a measure of uncertainty on its own.

Each step moves by $m/\sqrt{v}$ , times the learning rate. Dividing by $\sqrt{v}$ removes the gradient's scale: multiply every gradient by a constant $c>0$ and both $m$ and $\sqrt{v}$ grow by $c$ , so the step is unchanged. When recent gradients are nearly equal, $|m|$ is close to $\sqrt{v}$ and the step is close to the learning rate. When they keep flipping sign, $m$ averages toward zero while $v$ does not, and the step shrinks. Kingma and Ba describe the ratio as a signal-to-noise ratio.

Study prompts for this section: Intuition

Study prompts

Each button copies a prompt about this section to your clipboard, for an AI assistant you already use. Nothing is sent by this site.

Mathematics

Definitions, assumptions and the derivation.

Equation 1 Equation 2 Equation 3

Let $t=1,2,\ldots$ count updates from the starting parameters $\theta_0$ , and let $g_t = \nabla_\theta \mathcal{L}_t(\theta_{t-1})$ be the (stochastic) gradient for update $t$ , taken at the parameters before it.

Adam keeps exponential moving averages:

m_t = \beta_1 m_{t-1} + (1-\beta_1) g_t

v_t = \beta_2 v_{t-1} + (1-\beta_2) g_t^2

Because $m_t$ and $v_t$ start at zero, Adam uses bias correction:

\hat m_t = \frac{m_t}{1-\beta_1^t}, \qquad \hat v_t = \frac{v_t}{1-\beta_2^t}

Update rule:

\theta_t = \theta_{t-1} - \alpha \frac{\hat m_t}{\sqrt{\hat v_t} + \varepsilon}.

The division is elementwise, so Adam acts like a diagonal preconditioner: a coordinate with large recent gradients gets a smaller effective learning rate $\alpha/(\sqrt{\hat v_t}+\varepsilon)$ , and one with small gradients a larger one. The step is that rate times $\hat m_t$ , so large gradients do not mean small steps: when a coordinate's gradient stays about the same from step to step, $|\hat m_t|$ is close to $\sqrt{\hat v_t}$ and the coordinate moves by about $\alpha$ , whatever that gradient's size. A single large gradient is different: it keeps $\sqrt{\hat v_t}$ high for many steps and shrinks the steps that follow. Bias correction matters most early in training because the moving averages are initialized at zero. The small $\varepsilon$ is not a learning signal; it prevents division by zero and can affect stability when gradients are tiny.

Typical defaults: $\beta_1=0.9$ , $\beta_2=0.999$ , $\varepsilon=10^{-8}$ .

What bias correction divides out

Unrolling the first recurrence from $m_0=0$ shows exactly what the correction removes:

m_t=(1-\beta_1)\sum_{k=1}^{t}\beta_1^{t-k}g_k, \qquad \sum_{k=1}^{t}(1-\beta_1)\beta_1^{t-k}=1-\beta_1^t .

The weights on past gradients add up to $1-\beta_1^t$ , not to one, so early in training $m_t$ is an average shrunk towards zero. Dividing by $1-\beta_1^t$ makes the weights sum to one, and the same holds for $v_t$ with $\beta_2$ . Kingma and Ba (2015, Section 3) derive the correction as removing this initialization bias in expectation, assuming the gradient distribution changes slowly; it does not make every corrected value an unbiased estimate when the gradients drift.

Two steps by hand

Take two coordinates with $\beta_1=1/2$ , $\beta_2=7/8$ , $\alpha=1/10$ , $\varepsilon=0$ , $\theta_0=(0,0)$ , and gradients $g_1=(2,-8)$ and $g_2=(-2,-8)$ . The numbers are chosen so that every value in the two steps is a fraction. Coordinate by coordinate:

\begin{aligned} t=1:&\quad m_1=(1,-4),\ v_1=(\tfrac12,8),\ \hat m_1=(2,-8),\ \hat v_1=(4,64),\ \Delta\theta_1=(-\tfrac1{10},\tfrac1{10}),\\ t=2:&\quad m_2=(-\tfrac12,-6),\ v_2=(\tfrac{15}{16},15),\ \hat m_2=(-\tfrac23,-8),\ \hat v_2=(4,64),\ \Delta\theta_2=(\tfrac1{30},\tfrac1{10}). \end{aligned}

At $t=1$ both coordinates move by exactly $\alpha$ , although one gradient is four times the other: the first corrected step is $-\alpha\,g_1/|g_1|$ . Without the two corrections the first ratio would be $m_1/\sqrt{v_1}=(\sqrt2,-\sqrt2)$ , too large by the factor $(1-\beta_1)/\sqrt{1-\beta_2}=\sqrt2$ . With the usual $\beta_1=0.9$ and $\beta_2=0.999$ that factor is about $3.16$ at the first step, and it fades slowly: for a constant gradient the uncorrected step is $(1-\beta_1^t)/\sqrt{1-\beta_2^t}$ times the corrected one, still about $1.26$ after $1{,}000$ steps. At $t=2$ coordinate 1's gradient changes sign. Its first average partly cancels and its step shrinks to $\alpha/3$ , while its second average, built from squares, stays at $4$ . Coordinate 2's gradients agree, and it keeps moving by $\alpha$ .

Rescaling one coordinate

Multiply one coordinate's whole gradient history by a constant $c$ and keep everything else fixed. The recurrences are linear in $g$ and in $g^2$ , so for that coordinate

\hat m'_t=c\,\hat m_t, \qquad \hat v'_t=c^2\,\hat v_t, \qquad \frac{\hat m'_t}{\sqrt{\hat v'_t}+\varepsilon} =\frac{c\,\hat m_t}{|c|\sqrt{\hat v_t}+\varepsilon}.

When $c>0$ and $\varepsilon=0$ this is exactly the old ratio: with $c=10$ in the example above, both steps are unchanged. Each condition matters. With $\varepsilon>0$ the ratio becomes $\hat m_t/(\sqrt{\hat v_t}+\varepsilon/c)$ instead of $\hat m_t/(\sqrt{\hat v_t}+\varepsilon)$ : nearly the same while $\varepsilon$ is small next to both $\sqrt{\hat v_t}$ and $c\sqrt{\hat v_t}$ , but very different when the rescaled gradients become as small as $\varepsilon$ . With $c<0$ the step keeps its size and reverses its direction. With $c=0$ and $\varepsilon=0$ it is $0/0$ . In floating-point arithmetic the cancellation is exact only up to rounding: in double precision the second-step ratio of the rescaled coordinate is $-0.3333333333333333$ before rescaling and $-0.33333333333333337$ after, one unit apart in the last place. This is the invariance to gradient scale that Kingma and Ba (2015, Section 2.1) note; it describes the update rule, not whether Adam trains a given model well.

Study prompts for this section: Mathematics

Study prompts

Each button copies a prompt about this section to your clipboard, for an AI assistant you already use. Nothing is sent by this site.

Code

The same calculation as runnable code, in the notation of the derivation.

Code example 1import torch alpha = 0.1 beta1, beta2 = 0.9, 0.999 eps = 1e-8 theta = torch.tensor([3.0], req...python

import torch

alpha = 0.1
beta1, beta2 = 0.9, 0.999
eps = 1e-8

theta = torch.tensor([3.0], requires_grad=True)
m = torch.zeros_like(theta)
v = torch.zeros_like(theta)

for t in range(1, 51):
    loss = (theta ** 2).sum()
    loss.backward()

    g = theta.grad.detach()
    m = beta1 * m + (1 - beta1) * g
    v = beta2 * v + (1 - beta2) * (g * g)

    mhat = m / (1 - beta1 ** t)
    vhat = v / (1 - beta2 ** t)

    theta = (theta - alpha * mhat / (vhat.sqrt() + eps)).detach().requires_grad_(True)

    if t in [1, 5, 10, 50]:
        print(t, "theta=", theta.item(), "loss=", loss.item())

The second block repeats the two steps worked by hand above in exact fractions, using only Python's standard library. It checks each step, the $\sqrt2$ ratio without bias correction and the unchanged steps after rescaling, then computes the rescaled ratio again in ordinary floating point, where it differs in the last place.

from fractions import Fraction as F
from math import isqrt, sqrt

B1, B2, ALPHA = F(1, 2), F(7, 8), F(1, 10)


def exact_root(value):
    """Square root of a fraction that is a perfect square, as every value here is."""
    root = F(isqrt(value.numerator), isqrt(value.denominator))
    assert root * root == value
    return root


def adam_coordinate(gradients):
    """One coordinate of Adam with epsilon = 0: the update treats coordinates separately."""
    m = v = theta = F(0)
    steps = []
    for t, g in enumerate(map(F, gradients), start=1):
        m = B1 * m + (1 - B1) * g
        v = B2 * v + (1 - B2) * g * g
        m_hat, v_hat = m / (1 - B1**t), v / (1 - B2**t)
        step = -ALPHA * m_hat / exact_root(v_hat)
        theta += step
        steps.append({"m_hat": m_hat, "v_hat": v_hat, "step": step, "theta": theta})
    return steps


first = adam_coordinate([2, -2])     # coordinate 1: its gradient changes sign
second = adam_coordinate([-8, -8])   # coordinate 2: its gradients agree
assert [s["step"] for s in first] == [F(-1, 10), F(1, 30)]
assert [s["step"] for s in second] == [F(1, 10), F(1, 10)]
assert (first[1]["m_hat"], first[1]["v_hat"]) == (F(-2, 3), F(4))

# Without bias correction the first ratio is m_1 / sqrt(v_1) = 1 / sqrt(1/2) = sqrt(2).
m_1, v_1 = (1 - B1) * 2, (1 - B2) * 2**2
assert abs(m_1 / sqrt(v_1) - sqrt(2)) < 1e-15

# Multiplying coordinate 1's gradients by c = 10 leaves every step unchanged.
assert [s["step"] for s in adam_coordinate([20, -20])] == [s["step"] for s in first]


def float_ratio(gradients):
    """The same coordinate in ordinary double-precision arithmetic: the last ratio."""
    m = v = 0.0
    for t, g in enumerate(gradients, start=1):
        m = 0.5 * m + 0.5 * g
        v = 0.875 * v + 0.125 * g * g
        ratio = (m / (1 - 0.5**t)) / sqrt(v / (1 - 0.875**t))
    return ratio


print("coordinate 1 steps:", [str(s["step"]) for s in first])
print("coordinate 2 steps:", [str(s["step"]) for s in second])
print("second ratio, before and after c = 10:", float_ratio([2.0, -2.0]), float_ratio([20.0, -20.0]))

Study prompts for this section: Code

Study prompts

Each button copies a prompt about this section to your clipboard, for an AI assistant you already use. Nothing is sent by this site.

Demo

Change an input, predict the result, and see which quantities respond.

ExploreChange one control at a time and name what responds.An optional observation guide follows the demo.

Explore Adam Optimizer

Loading the interactive demo…

Observation guide

Optional and ungraded. Opening a guide saves nothing.

Choose what to inspect in Adam Optimizer, then open its guide.

The first code block and the demo are two different examples of the same per-coordinate update rule; they are not the same run. That code optimizes one parameter with $\mathcal L(\theta)=\theta^2$ , starting at $\theta=3$ with $\alpha=0.1$ . The demo races SGD, SGD with momentum and Adam on the two-parameter Rosenbrock loss $\mathcal L(x,y)=(1-x)^2+100(y-x^2)^2$ , whose minimum is $0$ at $(1,1)$ . All three start at $(-1.5,1.5)$ , use the exact gradient $\nabla\mathcal L=\big(-2(1-x)-400x(y-x^2),\;200(y-x^2)\big)$ with no mini-batch noise, and share one $\alpha$ . SGD updates $\theta\leftarrow\theta-\alpha\nabla\mathcal L$ ; the momentum runner reuses $\beta_1$ as $v\leftarrow\beta_1v-\alpha\nabla\mathcal L$ , then $\theta\leftarrow\theta+v$ .

What carries over between the two examples is the first step. At $t=1$ , bias correction gives $\hat m_1=g_1$ and $\hat v_1=g_1^2$ , so each coordinate changes by $\Delta\theta=-\alpha g_1/(|g_1|+\varepsilon)\approx-\alpha\operatorname{sign}(g_1)$ : a step of size about $\alpha$ whatever the scale of $g_1$ , as long as $|g_1|\gg\varepsilon$ . In that code, $g_1=6$ , so $\Delta\theta\approx-0.1$ and $\theta$ goes from $3$ to about $2.9$ ; the first printed line pairs that updated $\theta$ with the loss $9$ computed before the update. In the demo at $\alpha=0.01$ , $g_1=(-455,-150)$ . Adam's update is about $(+0.01,+0.01)$ , to about $(-1.49,1.51)$ , while SGD and momentum both take $-\alpha g_1=(+4.55,+1.50)$ to $(3.05,3.00)$ . The demo also adds $10^{-8}$ inside its bias-correction denominators, which changes this first Adam step by a relative amount of about $5\times10^{-6}$ at $\beta_2=0.999$ . With AdamW on, the demo also subtracts $\alpha\lambda\theta$ with $\lambda=0.05$ : decoupled weight decay, applied outside the adaptive step (Loshchilov and Hutter, 2019). For Adam this differs from adding an $L_2$ penalty to the loss, whose gradient would also be divided by $\sqrt{\hat v_t}$ .

Before the race, predict which optimizer has the lowest loss after 60 updates. A loss that is no longer a finite number is reported as diverged and cannot win, and equal lowest losses are reported as a tie. The result describes this one deterministic run, not a general ranking of optimizers; other scenarios in the demo change the order.

Study prompts for this section: Demo

Study prompts

Each button copies a prompt about this section to your clipboard, for an AI assistant you already use. Nothing is sent by this site.

Practice

Work out two exact steps, see what bias correction changes, then scale one coordinate's gradients by 10 and predict again.

Loading the exercise…

Focus on

In focus

Concept: Adam Optimizer

What is the smallest example of Adam Optimizer you can work through by hand, and what does it show?

BeforeDerivativesNextWeight Decay & AdamW: Decoupled Regularization

DetailsOptimization

About this itemOptimization

3 sources citedChecking this browser

Reference id

concept:optimization/adam

Math3 equationsm_t = \beta_1 m_{t-1} + (1-\beta_1) g_t CodeAdam Optimizer code example 1assert root * root == value DemoAdam Optimizer interactive demoChange one control at a time and name what responds.

After the first pass

Go deeper

Optional material for a second reading. Sources are always shown; the other panels open from their headings or from these links, and closing one keeps what you wrote or revealed.

What to look for in each sectionChoose a section and what you expect to see in it, then read short advice.

Section by section

Choose what to look for in each section

Adam combines momentum, a running average of gradients, with a per-parameter step size scaled by a running average of squared gradients.

Choose what to look for (no demo yet)01 / Intuition

Illustration of adaptive Adam steps using momentum and second-moment scaling across a loss surface.

What to look for

Start with the picture, metaphor, or geometric mechanism.

Choose first

Choose what you expect to see in this section of Adam Optimizer.

Questions to ask of the pictureChoose a part of the picture and what you expect, then read short advice.

Looking at the picture

Questions to ask of the picture

Adam combines momentum, a running average of gradients, with a per-parameter step size scaled by a running average of squared gradients.

4 of 4 sections writtenNo live demo yet

Visual cueWhich part of the picture should you look at first?

Highlight size2/4

Question

Which part of the picture should you look at first?

Choose first

Pick the part of the picture you expect to explain Adam Optimizer.

Sources

References for this page

A reference shows where an idea comes from; it does not vouch for every step on this page.

paper · 2015Adam: A Method for Stochastic OptimizationKingma and Ba

Why it is cited

Defines Adam's first- and second-moment estimates, their bias correction and the adaptive step (ICLR 2015). Section 2.1 discusses the step-size bound, the signal-to-noise ratio and invariance to gradient scale.

Open source

paper · 2019Decoupled Weight Decay RegularizationLoshchilov and Hutter

Why it is cited

Shows that L2 regularization and weight decay differ for Adam and proposes AdamW, which applies weight decay separately from the adaptive gradient step (ICLR 2019).

Open source

book · 2016Deep LearningGoodfellow, Bengio, and CourvilleSee Sections 8.3.2 and 8.5.3

Why it is cited

Textbook treatment of momentum (Section 8.3.2) and of Adam among adaptive learning-rate methods (Section 8.5.3).

Open source

What each claim rests on2 claims, each with its sources, where it appears on this page, and its caveats.

Claims and sources

Adam combines momentum, a running average of gradients, with a per-parameter step size scaled by a running average of squared gradients.

Each claim checked against the cited passages

The checks were made by this site, not by an independent reviewer.

Adam maintains exponential moving averages of gradients and squared gradients, bias-corrects both from zero initialization, and takes elementwise adaptive steps by subtracting alpha*mhat_t/(sqrt(vhat_t)+epsilon).

Sources

Adam: A Method for Stochastic OptimizationDefines Adam's first- and second-moment estimates, their bias correction and the adaptive step (ICLR 2015). Section 2.1 discusses the step-size bou...

What the sources support

Kingma and Ba's Algorithm 1 initializes m0 and v0 to zero, updates biased first-moment and second raw-moment estimates from gt and gt^2, computes bias-corrected estimates, and updates parameters with alpha*m...

Where to see it on this page

Adam: A Method for Stochastic Optimization Equation 1 Equation 2 Code example 1 Demo

Equation 1

m_t = \beta_1 m_{t-1} + (1-\beta_1) g_t

Equation 2

v_t = \beta_2 v_{t-1} + (1-\beta_2) g_t^2

Caveat

This checks Adam's original adaptive-moment update mechanics, not convergence guarantees, AdamW decoupled weight decay, AMSGrad, optimizer generalization debates, sparse-gradient variants, or framework-speci...

StatusChecked against the cited passages.

Checked Kingma and Ba Algorithm 1 plus sections 2 and 3: the paper initializes m0 and v0 as zero vectors, updates biased first and second raw moment estimates from gt and gt^2, divides by 1-beta1^t and 1-beta2^t for bias correction, states vector operations are element-wise, and updates theta with alpha*mhat/(sqrt(vhat)+epsilon). Scope is original Adam update mechanics only.

With epsilon = 0, multiplying one coordinate's whole gradient history by a constant c > 0 multiplies its corrected first moment by c and its corrected second moment by c squared, so Adam's step for that coordinate is unchanged.

Sources

Adam: A Method for Stochastic OptimizationDefines Adam's first- and second-moment estimates, their bias correction and the adaptive step (ICLR 2015). Section 2.1 discusses the step-size bou...

What the sources support

Kingma and Ba (Section 2.1) note that rescaling the gradients by a factor c scales mhat by c and vhat by c squared, which cancel in the step. The page's two-step example shows the cancellation in exact fract...

Where to see it on this page

Adam: A Method for Stochastic Optimization Equation 3

Equation 3

\hat m_t = \frac{m_t}{1-\beta_1^t}, \qquad \hat v_t = \frac{v_t}{1-\beta_2^t}

Caveat

Exact only for c > 0, epsilon = 0 and the whole gradient history; with epsilon > 0 the step can change, a lot once the scaled gradients are as small as epsilon; a negative c reverses it, and floating-point r...

StatusChecked against the cited passages.

Read Kingma and Ba Section 2.1, 'Adam's update rule' (with epsilon = 0 the effective step is invariant to the scale of the gradients: rescaling g by a factor c scales mhat by c and vhat by c squared, which cancel), and Section 3 (the bias-correction derivation, exact when the second moment is stationary). Recomputed the page's two-step example and the c = 10 rescaling in exact fractions and in double precision.

Explain it without the pageWrite your own explanation before asking for feedback. Your draft and any hints stay while this page is open.

Practice · Adam Optimizer

Try the idea in your own words

Adam combines momentum, a running average of gradients, with a per-parameter step size scaled by a running average of squared gradients.

Concept · Selected for practice

Adam Optimizer

What it rests on: Sources: Adam: A Method for Stochastic Optimization; Decoupled Weight Decay Regularization; Deep Learning

Context and links

Optimization

Choose a task

Explain the mechanism

For Adam Optimizer: What is the smallest example of Adam Optimizer you can work through by hand, and what does it show? Explain your answer, including what changes, why, and which assumption matters.

Your answerNo answer yet

A rough first thought is enough. Your draft stays when you change tasks.

Your draft stays on this page and is cleared when you leave or reload.

A little help · Explain

Open one hint at a time. These are suggestions, not your answer or a grade.

0 of 3 hints shown for this question.

Clearing your answer keeps this help record. Outside help is your own report; this page cannot check it.

Where am I stuck? (optional)

Your own description, not an automatic diagnosis

Choose one, or leave this unspecified. Select it again to clear it.

Take your draft to a feedback conversation

No AI feedback runs here. You can copy a prompt to use elsewhere; nothing is sent automatically. Review the text before sharing, and leave out private information.

Write an attempt before copying a feedback prompt.

A draft, or an AI reply to it, is not a test of what you have learned. To check that, try a different case later without help.

Ask about this pagePick one item on this page and copy a prompt about it into an AI assistant you already use. Notes you write here stay in this browser.

Ask about this pageClose

ConceptAdam OptimizerSources: Adam: A Method for Stochastic Optimization; Decoupled Weight Decay Regularization; Deep Learningassert root * root == value

Your question

Choose what your question is about

Pick the idea, equation, source, code, claim, misconception or demo state you want to ask about. The prompt you copy includes it, so the answer can stay on that item.

Next local actionNo local draft saved yet

Open the draft below to save one note and next action in this browser.

conceptOptimization

Adam Optimizer

Starting question

What is the smallest example of Adam Optimizer you can work through by hand, and what does it show?

SourcesCheck the 3 cited sources listed under Sources.Notes can be saved for this item

Points of view on this item

Fixed prompts generated from this item. They are not comments from people or independent reviews.

Learner’s first stepAsk what would make "Adam Optimizer" feel predictable rather than familiar.

Assumption

The 3 cited sources must support this exact item, not just the surrounding topic.

Source-checking summary

Connect the definition to one equation, piece of code or demo before widening the discussion.

Proposed experiment

Change one input, then check that the same relationship holds in the mathematics, the code and the demo.

Next action

You can state the mechanism in your own words

Evidence4 checks

ObservationChecking for a saved observation

ActionReady for one action

PromptLearner prompt ready to copy

Go to this item

01ObservationChecking what this browser has saved

02EvidenceChecking for a saved observation

03SourcesCheck the 3 cited sources listed under Sources.

04Next stepSave one next action

Local action draftNo local draft saved yetOpen when you are ready to write one next action

Local action draft

This draft stays in this browser and is attached to this item only.

Draft noteNext action

No local draft saved.

Evidence to inspect

What the 3 cited sources say about this exact item
The definition, its prerequisites, and a contrasting concept
The equation or code that makes the concept concrete
One demo state that shows what stays the same as the inputs change

What would resolve this

You can state the mechanism in your own words
You can name the prerequisite that would clear up the confusion
You can predict how the result changes when one input changes

Reference id

concept/concept-notebook/optimization/adam
concept:optimization/adam

Return to reading

Reading map

In this concept

Change one control at a time and name what responds.

Try the demo

Continue the curriculum

Part II · unit 2 of 6

Weight Decay & AdamW: Decoupled Regularization

Connections and sources

Domain

Optimization

optimization
deep-learning
adam

Prerequisites

Leads to

Study aid

Ask an assistant about this notebook

Compose a question with this notebook as context, then copy it into an AI assistant you already use.

Compose a question

Your question

GoalComfortStyleStuck on

Copy-only tool: this panel composes a prompt from this page and your choices above, for an AI assistant you already use. Nothing you enter here is sent from this page.