Ask what should change under a concrete input, then trace that expectation through the equation.
Foundation Lab
Activation Checkpointing & Memory Efficiency
Essential for training large models—without it, you can't fit 100B models in GPU memory
#79CheckpointingEfficiency
key equation
Selected Foundation Object
Keep the equation fixed; move through the evidence.
Concept 79 of 100CheckpointingEfficiency / Phase 6: Modern efficiency & inference
Current question
Backprop needs activations from forward pass; normally we store all of them
Use the key equation and canonical papers as the available witnesses, without implying that a runnable panel exists.
The useful learning product is the reusable mechanism you can carry into another model, paper, or engineering tradeoff.
Use prerequisites, dependents, and semantic links to repair the next gap without leaving the object behind.
Why It Matters for Modern Models
- Essential for training large models—without it, you can't fit 100B models in GPU memory
- Every major training framework (PyTorch, JAX) uses this technique
- Memory-compute trade-off is fundamental: pay with one to save the other
What Tutorials Skip
What is still poorly explained in textbooks and papers:
- Backprop needs activations from forward pass; normally we store all of them
- Checkpointing says: "just save some, recompute the rest when needed"
- Optimal checkpoint spacing is √L layers—minimizes memory × compute product
Visualization Status
Core Math (Optional Deep Dive)
If you want intuition first, start with the key equation and cited sources. Come back here for the full walkthrough.
Key Equation
Memory problem: Storing activations for backprop requires memory.
Gradient checkpointing: Recompute instead of store:
- Forward: Save only checkpoint activations (every layers)
- Backward: Recompute activations from nearest checkpoint
Memory: instead of
Compute: 33% overhead (recompute forward pass once)
Selective checkpointing: Only checkpoint expensive layers (attention).
Trade-off: