Domain
Optimization
How neural networks are trained: gradient steps, learning rates and schedules, curvature and sharpness, and adaptive optimizers such as Adam.
Preview the recommended route
Start here. Make one prediction, then note what carries over to the next notebook.
θ_t∇L(θ)updateθ_{t+1}LearnerWhich notebook should I read first?
Open the first notebookRead the first notebook, make one prediction in its demo, then go on to the next.
Recommended route
Start here, then follow the prerequisites forward.
Read these in order: each notebook comes after the notebooks in this domain that it builds on, and the label beside it gives its level. A prerequisite with no page here yet is marked as outside reading. A notebook is left out of the route only when a prerequisite has neither a page here nor a named outside reading; the full list below marks it.
- 01Gradient Descent
Gradient descent turns local slope information into an iterative update rule for reducing a loss.
Introductory20 mincodedemoafter Derivatives - 02Adam Optimizer
Adam combines momentum, a running average of gradients, with a per-parameter step size scaled by a running average of squared gradients.
Intermediate16 mincodedemoafter Derivatives, Gradient DescentAdam Optimizer builds directly on Gradient Descent.
- 03Loss Landscapes, Sharpness & Flat Minima
What 2D slices of a loss surface show and hide, how Hessian curvature measures sharpness, why the link between flat minima and generalization is contested, and how SAM trains toward flat regions.
Intermediate16 mincodedemoafter Adam OptimizerLoss Landscapes, Sharpness & Flat Minima builds directly on Adam Optimizer.
- 04Gradient Clipping & Explosion Prevention
Norm clipping rescales gradients above a threshold while preserving their direction. It bounds a plain gradient-descent step, but the same bound need not hold for an adaptive optimizer that rescales coordinates or averages earlier gradients.
Intermediate13 mincodedemoafter Backpropagation, Loss Landscapes, Sharpness & Flat MinimaGradient Clipping & Explosion Prevention builds directly on Loss Landscapes, Sharpness & Flat Minima.
- 05Learning Rate Schedules: Warmup, Decay & Cycling
How warmup, decay and cyclical schedules change the learning rate over a training run, including the Transformer's warmup with inverse-square-root decay and cosine annealing.
Intermediate14 mincodedemoafter Adam Optimizer, Loss Landscapes, Sharpness & Flat Minima - 06Weight Decay & AdamW: Decoupled Regularization
Why shrinking weights is not the same as adding an L2 penalty inside Adam, and how AdamW restores the intended regularization behavior.
Intermediate13 mincodedemoafter Adam Optimizer, Maximum Likelihood
All notebooks
Every published notebook in this domain.
Gradient Descent
Gradient descent turns local slope information into an iterative update rule for reducing a loss.
Adam Optimizer
Adam combines momentum, a running average of gradients, with a per-parameter step size scaled by a running average of squared gradients.
Loss Landscapes, Sharpness & Flat Minima
What 2D slices of a loss surface show and hide, how Hessian curvature measures sharpness, why the link between flat minima and generalization is contested, and how SAM trains toward flat regions.
Gradient Clipping & Explosion Prevention
Norm clipping rescales gradients above a threshold while preserving their direction. It bounds a plain gradient-descent step, but the same bound need not hold for an adaptive optimizer that rescales coordinates or averages earlier gradients.
Learning Rate Schedules: Warmup, Decay & Cycling
How warmup, decay and cyclical schedules change the learning rate over a training run, including the Transformer's warmup with inverse-square-root decay and cosine annealing.
Weight Decay & AdamW: Decoupled Regularization
Why shrinking weights is not the same as adding an L2 penalty inside Adam, and how AdamW restores the intended regularization behavior.