3Optimization

∇Adam & Adaptive Gradient Methods

Canonical Papers

Kingma & Ba2014ICLR

Reddi et al.2018ICLR

For gradient $g_t = \nabla_\theta L_t(\theta_t)$ :

\begin{aligned} m_t &= \beta_1 m_{t-1} + (1-\beta_1) g_t \\ v_t &= \beta_2 v_{t-1} + (1-\beta_2) g_t^2 \\ \hat m_t &= m_t / (1-\beta_1^t),\quad \hat v_t = v_t / (1-\beta_2^t) \\ \theta_{t+1} &= \theta_t - \alpha \frac{\hat m_t}{\sqrt{\hat v_t} + \varepsilon} \end{aligned}

Convergence analyses show that naïve Adam can diverge on simple convex problems and motivate variants like AMSGrad.

Key Equation

\theta_{t+1} = \theta_t - \alpha \frac{\hat m_t}{\sqrt{\hat v_t} + \varepsilon}

Large foundation models almost universally use Adam or AdamW for pretraining and fine-tuning
RLHF and diffusion training use Adam-style optimizers to handle noisy gradients and widely varying scales

What is still poorly explained in textbooks and papers:

Geometric explanation of how per-coordinate scaling with 1/√vₜ interacts with overparameterized nets—why it sometimes hurts generalization vs SGD
How Adam bias-correction and exponential averaging interact with curriculum and non-stationary objectives (e.g. RLHF)

Explore this concept from different angles — like a mathematician would.

⚠️ Breaks When

🔄 Same Technique

🔧 Invented to Fix