Bring the mental model from Overparameterization & Generalization (Double Descent); this page will reuse it instead of restarting from zero.
Scaling
Neural Tangent Kernel (NTK) & Infinite-Width Limits
A limit where wide neural networks behave like kernel methods: the model linearizes around initialization and training becomes kernel regression.
Concept Structure
Neural Tangent Kernel (NTK) & Infinite-Width Limits
01Intuition
Start with the picture, metaphor, or geometric mechanism.
02Math
Make the objects explicit and connect them with notation.
03Code
Mirror the equations with runnable implementation details.
04Interactive Demo
Manipulate the mechanism and watch the idea respond.
1prerequisites
1next concepts
2related links
Learning map
Neural Tangent Kernel (NTK) & Infinite-Width LimitsBeforeOverparameterization & Generalization (Double Descent)Now4/4 sections readyTryManipulate one control and predict the visible change.NextScaling Laws & Emergent Abilities
Conceptual Bridge
What should feel connected as you move through this page.
A limit where wide neural networks behave like kernel methods: the model linearizes around initialization and training becomes kernel regression.
The next edge should feel earned: use the demo prediction here before following Scaling Laws & Emergent Abilities.
Test the linkManipulate one control and predict the visible change.Then continue to Scaling Laws & Emergent Abilities
01IntuitionStart with the picture, metaphor, or geometric mechanism.02MathMake the objects explicit and connect them with notation.03CodeMirror the equations with runnable implementation details.04Interactive DemoManipulate the mechanism and watch the idea respond.
01
01
Intuition
Build the mental picture first so the rest of the page has something to attach to.
Section prompt
Neural networks can learn features: the representation changes during training.
The NTK perspective asks a different question:
What if the network is so wide that, during training, it barely moves in parameter space?
If parameters move only a little, the network behaves like its own first-order Taylor expansion around initialization. In that regime, training can often be analyzed as linearized dynamics rather than nonlinear feature learning. It becomes a kernel method in disguise, where the "features" are the gradients of the network output with respect to its parameters.
This is useful because kernel methods are mathematically tractable: you can often predict learning dynamics, and in some studied settings reason about generalization-related behavior, without simulating full training.
02
02
Math
Translate the story into symbols, assumptions, and a derivation you can inspect.
Section prompt
Equation 1\Theta(x,x') = \nabla_\theta f_\theta(x)^\top \nabla_\theta f_\theta(x').Equation 2\partial_t f_t(x) = -\sum_i \Theta(x,x_i)\,(f_t(x_i) - y_i),
Let be a network and its parameters. The neural tangent kernel is:
Consider gradient flow on a squared loss over training points , with .
In the infinite-width NTK limit for the standard parameterizations studied in this literature, becomes deterministic and stays approximately constant during training. The function evolves according to:
which is a linear ODE in function space. The solution corresponds to kernel regression with kernel (with details depending on loss, step size, and regularization).
03
03
Code
Keep the implementation aligned with the notation so the algorithm is legible.
Section prompt
import numpy as np
rs = np.random.RandomState(0)
m = 128 # "width"
w = rs.randn(m)
v = rs.randn(m)
def grad_f(x):
z = w * x
s = (z > 0).astype(float)
g_v = np.maximum(z, 0) / np.sqrt(m)
g_w = (v * s * x) / np.sqrt(m)
return np.concatenate([g_v, g_w])
def K(x, xp):
g1, g2 = grad_f(x), grad_f(xp)
return float(g1 @ g2)
xtr = np.linspace(-1, 1, 40)
ytr = np.sin(3 * xtr)
Ktr = np.array([[K(x, xp) for xp in xtr] for x in xtr])
lam = 1e-3
alpha = np.linalg.solve(Ktr + lam * np.eye(len(xtr)), ytr)
xte = np.linspace(-1, 1, 200)
Kte = np.array([[K(x, xp) for xp in xtr] for x in xte])
pred = Kte @ alpha
mse = np.mean((pred - np.sin(3 * xte)) ** 2)
print("test MSE:", round(float(mse), 4))
04
04
Interactive Demo
Use direct manipulation to connect the explanation to a moving system.
Section prompt
Use the demo to compare:
- kernel-like learning dynamics (fixed features / linearized training),
- feature learning (representations change), and
- how width and fixed-feature assumptions push you toward one regime or the other.
Live Concept Demo
Explore Neural Tangent Kernel (NTK) & Infinite-Width Limits
The stage is code-native and interactive. Use it to test the explanation against the mechanism.
difficulty 4/5graduatecode-aligned
Demo Prediction Checkpoint
Manipulate one control and predict the visible change.
Commit to what Neural Tangent Kernel (NTK) & Infinite-Width Limits should make visible before reading the result.
After The First Pass
Turn the concept into an inspected object.
Once the invariant is visible in the intuition, math, code, and demo, use these panels to inspect the mechanism visually, check source support, practice the idea, and attach a grounded research question.
Mechanism Storyboard
See the idea move before the page explains it
A limit where wide neural networks behave like kernel methods: the model linearizes around initialization and training becomes kernel regression.
Prediction open01 / Intuition
Prediction lens
Start with the picture, metaphor, or geometric mechanism.
Commit first
Before reading further, choose the kind of change Neural Tangent Kernel (NTK) & Infinite-Width Limits should make visible.
Visual Inquiry
Make the image answer a mathematical question
A limit where wide neural networks behave like kernel methods: the model linearizes around initialization and training becomes kernel regression.
4/4 stages readyLive demo connected
Visual cueWhich visible object should carry the first intuition?
Prediction
Which visible object should carry the first intuition?
Commit first
Pick the cue that should make Neural Tangent Kernel (NTK) & Infinite-Width Limits easier to reason about before the page gives the answer.
Source Grounding
Canonical references for the mechanism on this page.
Introduces the neural tangent kernel as the infinite-width kernel governing gradient-descent dynamics.
Open sourceGrounds the linearized-training view of wide neural networks near initialization.
Open sourceClaim Review
A limit where wide neural networks behave like kernel methods: the model linearizes around initialization and training becomes kernel regression.
Source IDs and witness objects are attached for review; they are not proof by themselves.
jacot-2018-ntk, lee-2019-wide-networks-linear
Use equation, code, and demo objects to check whether the source support is operational.
Jacot et al. introduce the NTK as the parameter-gradient inner-product kernel governing infinite-width gradient-flow dynamics. Lee et al. support the near-initialization linearized-training view for sufficiently wide networks. Local math shows the kernel definition and function-space ODE; code/demo illustrate fixed-gradient kernel behavior.
Sources: Neural Tangent Kernel: Convergence and Generalization in Neural Networks, Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient DescentThis does not certify finite-width feature learning, all architectures or parameterizations, all losses, all optimizers, or generalization guarantees. The checked claim is scoped to infinite-width or explicitly linearized wide-network NTK dynamics near initialization.Attached source IDs and witness refs are review targets, not proof.Source support candidates
paper 2018Neural Tangent Kernel: Convergence and Generalization in Neural NetworksIntroduces the neural tangent kernel as the infinite-width kernel governing gradient-descent dynamics.
paper 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient DescentGrounds the linearized-training view of wide neural networks near initialization.
Practice Loop
Try the idea before it explains itself
A limit where wide neural networks behave like kernel methods: the model linearizes around initialization and training becomes kernel regression.
Readiness0/3 checks ready
Predict
Before touching the demo, predict one visible change that should happen in Neural Tangent Kernel (NTK) & Infinite-Width Limits.
Hint 1
Reveal when your model needs a nudge.
Hint 2
Reveal when your model needs a nudge.
Hint 3
Reveal when your model needs a nudge.
Local checks
Claim
A concrete answer is on the canvas.
Mechanism
The answer names why the claim should hold.
Bridge
It touches the page context or a neighboring idea.
Object research drawerClose
ConceptNeural Tangent Kernel (NTK) & Infinite-Width LimitsScaling
Code witness comparisonNeural Tangent Kernel (NTK) & Infinite-Width Limits code witness 1rs = np.random.RandomState(0)Prediction before revealNeural Tangent Kernel (NTK) & Infinite-Width Limits interactive demoManipulate one control and predict the visible change.
Grounded room questionWhat is the smallest example that makes Neural Tangent Kernel (NTK) & Infinite-Width Limits click without losing the math?Local snapshot ready
Research Room
Attach the question to an exact object
Pick the concept, equation, source, code witness, claim, misconception, or demo state before asking for help. The handoff stays grounded to that object.Open the draft below to save one note and next action in this browser.
Neural Tangent Kernel (NTK) & Infinite-Width Limits
Anchored question
What is the smallest example that makes Neural Tangent Kernel (NTK) & Infinite-Width Limits click without losing the math?
Local action draftNo local draft saved yetExpand only when ready to capture one local next action
Local action draft
This draft stays locally in this browser for concept:scaling/ntk.
No local draft saved.
- Source ids to inspect: jacot-2018-ntk, lee-2019-wide-networks-linear
- Definition, prerequisite, and contrast concept links
- The equation or code witness that makes the concept operational
- One demo state that shows the invariant instead of a slogan
- The learner can state the mechanism in their own words
- The learner can name the prerequisite that would repair confusion
- The learner can predict how the mechanism changes under one perturbation
Grounded AI handoff
I am working in Continuous Function's research reading room. Object: concept - Neural Tangent Kernel (NTK) & Infinite-Width Limits Object key: concept:scaling/ntk Context: Scaling Anchor id: concept/concept-notebook/scaling/ntk Open question: What is the smallest example that makes Neural Tangent Kernel (NTK) & Infinite-Width Limits click without losing the math? Evidence to inspect: - Source ids to inspect: jacot-2018-ntk, lee-2019-wide-networks-linear - Definition, prerequisite, and contrast concept links - The equation or code witness that makes the concept operational - One demo state that shows the invariant instead of a slogan What would resolve this: - The learner can state the mechanism in their own words - The learner can name the prerequisite that would repair confusion - The learner can predict how the mechanism changes under one perturbation Answer as a careful research tutor: stay source-grounded, separate verified evidence from assumptions, name the relevant math objects, and end with one next action.
concept/concept-notebook/scaling/ntk
concept:scaling/ntk