Ask what should change when the equation is manipulated, then let the visualization test that expectation.
Foundation Lab
Adam & Adaptive Gradient Methods
Large foundation models almost universally use Adam or AdamW for pretraining and fine-tuning
#3AdamOptimization
key equation
\theta_{t+1} = \theta_t - \alpha \frac{\hat m_t}{\sqrt{\hat v_t} + \varepsilon}Selected Foundation Object
Keep the equation fixed; move through the evidence.
Concept 3 of 100AdamOptimization / Phase 3: Optimization & generalization
Current question
Geometric explanation of how per-coordinate scaling with 1/√vₜ interacts with overparameterized nets—why it sometimes hurts generalization vs SGD
\theta_{t+1} = \theta_t - \alpha \frac{\hat m_t}{\sqrt{\hat v_t} + \varepsilon}Use the runnable panel, the key equation, and canonical papers as separate forms of evidence for the same object.
The useful learning product is the reusable mechanism you can carry into another model, paper, or engineering tradeoff.
This atlas page keeps the working demo; the domain notebook carries the fuller Intuition -> Math -> Code -> Demo sequence.
Why It Matters for Modern Models
- Large foundation models almost universally use Adam or AdamW for pretraining and fine-tuning
- RLHF and diffusion training use Adam-style optimizers to handle noisy gradients and widely varying scales
What Tutorials Skip
What is still poorly explained in textbooks and papers:
- Geometric explanation of how per-coordinate scaling with 1/√vₜ interacts with overparameterized nets—why it sometimes hurts generalization vs SGD
- How Adam bias-correction and exponential averaging interact with curriculum and non-stationary objectives (e.g. RLHF)
Interactive Visualization
Core Math (Optional Deep Dive)
If you want intuition first, start with the key equation and the visualization. Come back here for the full walkthrough.
Key Equation
For gradient :
Convergence analyses show that naïve Adam can diverge on simple convex problems and motivate variants like AMSGrad.