Ask what should change when the equation is manipulated, then let the visualization test that expectation.
Foundation Lab
Neural Tangent Kernel & Infinite-Width Limits
NTK provides a mathematically clean limit where we can predict learning dynamics and generalization
#6NTKTheory
key equation
\Theta(x,x') = \nabla_\theta f_\theta(x)^\top \nabla_\theta f_\theta(x')Selected Foundation Object
Keep the equation fixed; move through the evidence.
Concept 6 of 100NTKTheory / Phase 3: Optimization & generalization
Current question
Most expositions are algebraic; missing is a geometric animation showing how trajectories in function space under NTK differ from genuine feature learning
\Theta(x,x') = \nabla_\theta f_\theta(x)^\top \nabla_\theta f_\theta(x')Use the runnable panel, the key equation, and canonical papers as separate forms of evidence for the same object.
The useful learning product is the reusable mechanism you can carry into another model, paper, or engineering tradeoff.
This atlas page keeps the working demo; the domain notebook carries the fuller Intuition -> Math -> Code -> Demo sequence.
Why It Matters for Modern Models
- NTK provides a mathematically clean limit where we can predict learning dynamics and generalization
- Many mechanistic-interpretability arguments assume behavior "somewhere between" kernel-like and feature-learning regimes
What Tutorials Skip
What is still poorly explained in textbooks and papers:
- Most expositions are algebraic; missing is a geometric animation showing how trajectories in function space under NTK differ from genuine feature learning
Interactive Visualization
Core Math (Optional Deep Dive)
If you want intuition first, start with the key equation and the visualization. Come back here for the full walkthrough.
Key Equation
Define network with parameters . The NTK is:
In the infinite-width limit, this kernel becomes deterministic and remains constant during training. Training becomes:
a linear ODE in function space, just like kernel regression.