The neural tangent kernel is the inner product of a network's parameter gradients at two inputs. In the infinite-width limit under the NTK parameterization, it stays fixed during training. With squared loss and gradient-flow training, the network then follows linear dynamics that can be solved in closed form; as training time grows, it converges to kernel regression with this kernel, plus a term from the function at initialization.
A newer notebook covers this concept. This older page keeps its summary, math and sources.
Concept 6 of 100NTKTheory · Study phase 3: Optimization & generalization
Question
How do training trajectories in function space differ between the NTK regime, where the kernel stays fixed, and feature learning, where the kernel itself changes?
How to study this pagePredict · Check · Generalize · Continue
PredictSay what you expect.
Before you move a control in the demo, say what you expect to change.
CheckCheck against the math and the papers.
Compare what the demo shows with the key equation and with the papers cited below.
GeneralizeName what stays the same.
Identify the part of the idea that survives a change of notation or model, so you can recognize it in another paper or system.
ContinueContinue in the newer notebook.
The newer notebook covers the same concept and has an interactive demo.
Why It Matters for Modern Models
The neural tangent kernel is the inner product of a network's parameter gradients at two inputs. In the infinite-width limit under the NTK parameterization, it stays fixed during training. With squared loss and gradient-flow training, the network then follows linear dynamics that can be solved in closed form; as training time grows, it converges to kernel regression with this kernel, plus a term from the function at initialization.
It is also a reference point for what real networks do differently: in this "lazy" regime the features barely move, and Chizat et al. (2019) found that convolutional networks forced to train this way performed worse on image tasks, which suggests that their success depends on features changing during training.
Interactive Demo
Core Math
If you prefer intuition first, try the demo above, then come back for the math.
Key Equation
Θ(x,x′)=∇θfθ(x)⊤∇θfθ(x′)
Define a scalar-output network fθ(x) with parameters θ. The neural tangent kernel (NTK) is:
Θ(x,x′)=∇θfθ(x)⊤∇θfθ(x′)
In the infinite-width limit (with the NTK parameterization), this kernel is deterministic at initialization and stays constant during training. Gradient-flow training then becomes:
∂tft(x)=−i∑Θ(x,xi)∂f∂ℓ(ft(xi),yi)
For squared loss this is a linear ODE in function space; as training time grows, the network converges to the kernel-regression solution with kernel Θ, plus a correction from the function at initialization (Jacot et al., 2018).
Key Papers and Books
Neural Tangent Kernel: Convergence and Generalization in Neural Networks
Feature learning dominates: where NTK breaks down, taken up in Circuits. Open Circuits
NTK assumes training stays near initialization (kernel/linearized regime). Induction heads emerge from representation reorganization and new circuit formation—behavior NTK-style predictions miss.
Fixed features suffice: where NTK breaks down, taken up in Double Descent. Open Double Descent
Double descent does not need feature learning: it already appears in random-feature models, whose features are fixed (Belkin et al., 2019), and so in the lazy regime that NTK describes. What the NTK picture misses is the learning of new features, which deep networks also do.
Where else does the same idea appear?
First-order Taylor: the same technique appears in Circuit Discovery. Open Circuit Discovery
NTK analyzes training via linearization around parameters; attribution patching estimates causal effects via first-order Taylor expansion around activations. Same approximation move: replace nonlinear change with a linear surrogate.
Linear lens: an analogy between NTK and Probing. Open Probing
NTK emphasizes what happens when learning is effectively linearized (features don't move much). Linear probing tests what a fixed representation already makes linearly accessible—both separate "linear extractability" from "feature learning."
Which ideas are its mathematical duals?
Width ↔ Depth: Scaling Laws is its mathematical dual. Open Scaling Laws
NTK theory describes networks in the limit of infinite width. Scaling laws are empirical: Kaplan et al. (2020) found that, at a fixed number of parameters, loss depends only weakly on the balance between width and depth. Both ask how the shape of a network matters at a given size.