Ask what should change when the equation is manipulated, then let the visualization test that expectation.
Foundation Lab
Dropout: Stochastic Regularization
Classic regularizer that prevents co-adaptation of features
#52DropoutOptimization
key equation
\tilde{h} = h \odot m, \quad m \sim \text{Bernoulli}(1-p)Selected Foundation Object
Keep the equation fixed; move through the evidence.
Concept 52 of 100DropoutOptimization / Phase 10: Mathematical foundations & information geometry
Current question
Dropout injects noise proportional to activation magnitude—implicitly favors robust features
\tilde{h} = h \odot m, \quad m \sim \text{Bernoulli}(1-p)Use the runnable panel, the key equation, and canonical papers as separate forms of evidence for the same object.
The useful learning product is the reusable mechanism you can carry into another model, paper, or engineering tradeoff.
Use prerequisites, dependents, and semantic links to repair the next gap without leaving the object behind.
Why It Matters for Modern Models
- Classic regularizer that prevents co-adaptation of features
- Foundation for understanding stochastic regularization (also: droppath, stochastic depth)
- Modern LLMs often use minimal dropout—understanding when it helps/hurts is practical knowledge
What Tutorials Skip
What is still poorly explained in textbooks and papers:
- Dropout injects noise proportional to activation magnitude—implicitly favors robust features
- Different dropout rates per layer: higher dropout in final layers often helps
- At large scale with lots of data, dropout can hurt: the ensemble benefit is dominated by data diversity
Interactive Visualization
Core Math (Optional Deep Dive)
If you want intuition first, start with the key equation and the visualization. Come back here for the full walkthrough.
Key Equation
Dropout randomly zeros activations during training:
At test time, scale by keep probability:
Or use inverted dropout (scale during training):
Dropout approximates ensemble averaging over sub-networks.