Continuous Function

Neural Tangent Kernel (NTK) & Infinite-Width Limits

In the infinite-width limit under the NTK parameterization, squared-loss training by gradient flow or with a sufficiently small learning rate follows the network's linearization at initialization. When the training kernel is invertible, the converged predictions add a kernel-regression fit of the initial errors to the initial predictions. The notebook contrasts this fixed-kernel regime with feature learning.

Advanced · Graduate mathematics · about 18 minutes

Reading map and next steps

Intuition

The idea in plain words, before the symbols.

Neural networks can learn features: the representation changes during training.

The NTK perspective asks a different question:

What if the network is so wide that, during training, it barely moves in parameter space?

If parameters move only a little, the network behaves like its own first-order Taylor expansion around initialization. In that regime, training can be analyzed as linearized dynamics rather than nonlinear feature learning (Jacot et al., 2018; Lee et al., 2019). It becomes a kernel method in disguise, where the "features" are the gradients of the network output with respect to its parameters.

This is useful because kernel methods are mathematically tractable: you can often predict learning dynamics, and in some settings reason about generalization, without simulating full training. The catch is that real networks often leave this regime. Chizat, Oyallon and Bach (2019) show that this "lazy" behaviour comes from how the model is scaled, and that the deep convolutional networks they tested classified images less accurately when trained in the lazy regime than with standard training, which does change the features.

Study prompts for this section: Intuition
Study prompts

Each button copies a prompt about this section to your clipboard, for an AI assistant you already use. Nothing is sent by this site.

Mathematics

Definitions, assumptions and the derivation.

Equation 1Equation 2

Let fθ(x)f_\theta(x) be a scalar-output network and θ\theta its parameters. The neural tangent kernel is:

Θ(x,x′)=∇θfθ(x)⊤∇θfθ(x′).\Theta(x,x') = \nabla_\theta f_\theta(x)^\top \nabla_\theta f_\theta(x').

Consider gradient flow on a squared loss over training points (xi,yi)(x_i,y_i), with L(θ)=12∑i(fθ(xi)−yi)2\mathcal L(\theta) = \tfrac12\sum_i (f_\theta(x_i) - y_i)^2.

In the infinite-width limit under the NTK parameterization of Jacot et al. (2018), trained by gradient flow or with a small enough learning rate (Lee et al., 2019), Θ\Theta becomes deterministic at initialization and stays constant during training. The function then evolves according to:

∂tft(x)=−∑iΘ(x,xi) (ft(xi)−yi),\partial_t f_t(x) = -\sum_i \Theta(x,x_i)\,(f_t(x_i) - y_i),

which is a linear ODE in function space. On the training inputs XX its solution is ft(X)−y=e−t Θ(X,X)(f0(X)−y)f_t(X)-y=e^{-t\,\Theta(X,X)}\big(f_0(X)-y\big), and, when Θ(X,X)\Theta(X,X) is invertible, as t→∞t\to\infty the prediction at any input becomes f∞(x)=f0(x)+Θ(x,X) Θ(X,X)−1(y−f0(X))f_\infty(x)=f_0(x)+\Theta(x,X)\,\Theta(X,X)^{-1}\big(y-f_0(X)\big): kernel regression with kernel Θ\Theta on what the initial network got wrong.

Study prompts for this section: Mathematics
Study prompts

Each button copies a prompt about this section to your clipboard, for an AI assistant you already use. Nothing is sent by this site.

Code

The same calculation as runnable code, in the notation of the derivation.

Code example 1import numpy as np rs = np.random.RandomState(0) m = 128 # width w, b, v = rs.randn(m), rs.ra...python
import numpy as np

rs = np.random.RandomState(0)
m = 128  # width
w, b, v = rs.randn(m), rs.randn(m), rs.randn(m)

# f(x) = sum_j v_j * relu(w_j * x + b_j) / sqrt(m); its parameter gradient is the NTK feature map
def grad_f(x):
    z = w * x + b
    s = (z > 0).astype(float)
    g_v = np.maximum(z, 0) / np.sqrt(m)
    g_w = (v * s * x) / np.sqrt(m)
    g_b = (v * s) / np.sqrt(m)
    return np.concatenate([g_v, g_w, g_b])

def K(x, xp):
    g1, g2 = grad_f(x), grad_f(xp)
    return float(g1 @ g2)

xtr = np.linspace(-1, 1, 40)
ytr = np.sin(3 * xtr)
Ktr = np.array([[K(x, xp) for xp in xtr] for x in xtr])

lam = 1e-3
alpha = np.linalg.solve(Ktr + lam * np.eye(len(xtr)), ytr)

xte = np.linspace(-1, 1, 200)
Kte = np.array([[K(x, xp) for xp in xtr] for x in xte])
pred = Kte @ alpha
mse = np.mean((pred - np.sin(3 * xte)) ** 2)
print("test MSE:", round(float(mse), 4))

The code builds the empirical kernel of a width-128 network at initialization and fits kernel ridge regression with it, leaving out the f0f_0 term for simplicity. It prints a test error of about 0.0010.001 on sin⁡(3x)\sin(3x). The bias terms matter: without them every gradient feature is a multiple of xx on each side of zero, the kernel has rank 2, and the fit can only be two straight lines meeting at the origin.

Study prompts for this section: Code
Study prompts

Each button copies a prompt about this section to your clipboard, for an AI assistant you already use. Nothing is sent by this site.

Demo

Change an input, predict the result, and see which quantities respond.

ExploreChange one control at a time and name what responds.An optional observation guide follows the demo.

Explore Neural Tangent Kernel (NTK) & Infinite-Width Limits

Loading the interactive demo…

Observation guide

Optional and ungraded. Opening a guide saves nothing.

Choose what to inspect in Neural Tangent Kernel (NTK) & Infinite-Width Limits, then open its guide.

Use the demo to compare:

  • kernel-like learning dynamics (fixed features / linearized training),
  • feature learning (representations change), and
  • how width and fixed-feature assumptions push you toward one regime or the other.
Study prompts for this section: Demo
Study prompts

Each button copies a prompt about this section to your clipboard, for an AI assistant you already use. Nothing is sent by this site.

In focus

Concept: Neural Tangent Kernel (NTK) & Infinite-Width Limits

What is the smallest example of Neural Tangent Kernel (NTK) & Infinite-Width Limits you can work through by hand, and what does it show?

BeforeOverparameterization & Generalization (Double Descent)NextTest-Time Compute: Spending Inference Budget on Search
DetailsScaling
About this itemScaling
3 sources citedChecking this browser
Reference idconcept:scaling/ntk
Math2 equations\Theta(x,x') = \nabla_\theta f_\theta(x)^T \nabla_\theta f_\theta(x').CodeNeural Tangent Kernel (NTK) & Infinite-Width Limits code example 1rs = np.random.RandomState(0)DemoNeural Tangent Kernel (NTK) & Infinite-Width Limits interactive demoChange one control at a time and name what responds.

After the first pass

Go deeper

Optional material for a second reading. Sources are always shown; the other panels open from their headings or from these links, and closing one keeps what you wrote or revealed.
What to look for in each sectionChoose a section and what you expect to see in it, then read short advice.

Section by section

Choose what to look for in each section

In the infinite-width limit under the NTK parameterization, squared-loss training by gradient flow or with a sufficiently small learning rate follows the network's linearization at initialization. When the training kernel is invertible, the converged predictions add a kernel-regression fit of the initial errors to the initial predictions. The notebook contrasts this fixed-kernel regime with feature learning.

Choose what to look for (no demo yet)01 / Intuition
Illustration of a neural tangent kernel matrix, infinite-width curve, and function-space dynamics.
What to look for

Start with the picture, metaphor, or geometric mechanism.

Choose first

Choose what you expect to see in this section of Neural Tangent Kernel (NTK) & Infinite-Width Limits.

Questions to ask of the pictureChoose a part of the picture and what you expect, then read short advice.

Looking at the picture

Questions to ask of the picture

In the infinite-width limit under the NTK parameterization, squared-loss training by gradient flow or with a sufficiently small learning rate follows the network's linearization at initialization. When the training kernel is invertible, the converged predictions add a kernel-regression fit of the initial errors to the initial predictions. The notebook contrasts this fixed-kernel regime with feature learning.

4 of 4 sections writtenNo live demo yet
Question

Which part of the picture should you look at first?

Choose first

Pick the part of the picture you expect to explain Neural Tangent Kernel (NTK) & Infinite-Width Limits.

Sources

References for this page

A reference shows where an idea comes from; it does not vouch for every step on this page.
paper · 2018Neural Tangent Kernel: Convergence and Generalization in Neural NetworksJacot, Gabriel, and Hongler
Why it is cited

Introduces the neural tangent kernel and shows that, in the infinite-width limit, it is deterministic at initialization and constant during gradient-flow training (NeurIPS 2018).

Open source
paper · 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient DescentLee et al.
Why it is cited

Shows that sufficiently wide networks trained by gradient descent with a small enough learning rate stay close to their first-order Taylor expansion around initialization, and gives the resulting closed-form predictions (NeurIPS 2019).

Open source
paper · 2019On Lazy Training in Differentiable ProgrammingChizat, Oyallon, and Bach
Why it is cited

Shows that the linearized (lazy) regime is caused by the model's scaling, and reports that lazy training often performs worse than standard training of convolutional networks (NeurIPS 2019).

Open source
What each claim rests on1 claim, each with its sources, where it appears on this page, and its caveats.

Claims and sources

In the infinite-width limit under the NTK parameterization, squared-loss training by gradient flow or with a sufficiently small learning rate follows the network's linearization at initialization. When the training kernel is invertible, the converged predictions add a kernel-regression fit of the initial errors to the initial predictions. The notebook contrasts this fixed-kernel regime with feature learning.

Checked against the cited passages

The checks were made by this site, not by an independent reviewer.

In infinite-width NTK limits, or when wide-network training is well approximated by linearization near initialization, the parameter-gradient inner-product NTK governs squared-loss gradient-descent function dynamics like a kernel method.
Sources
Neural Tangent Kernel: Convergence and Generalization in Neural NetworksIntroduces the neural tangent kernel and shows that, in the infinite-width limit, it is deterministic at initialization and constant during gradien...Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient DescentShows that sufficiently wide networks trained by gradient descent with a small enough learning rate stay close to their first-order Taylor expansio...
What the sources support

Jacot et al. introduce the NTK as the parameter-gradient inner-product kernel governing infinite-width gradient-flow dynamics. Lee et al. support the near-initialization linearized-training view for sufficie...

Where to see it on this page
Neural Tangent Kernel: Convergence and Generalization in Neural NetworksWide Neural Networks of Any Depth Evolve as Linear Models Under Gradient DescentEquation 1Equation 2Code example 1Demo
Equation 1
Θ(x,x′)=∇θfθ(x)⊤∇θfθ(x′).\Theta(x,x') = \nabla_\theta f_\theta(x)^\top \nabla_\theta f_\theta(x').
Equation 2
∂tft(x)=−∑iΘ(x,xi) (ft(xi)−yi),\partial_t f_t(x) = -\sum_i \Theta(x,x_i)\,(f_t(x_i) - y_i),
Code example 1import numpy as np rs = np.random.RandomState(0) m = 128 # width w, b, v = rs.randn(m), rs.ra...
Caveat

This describes infinite-width or explicitly linearized wide networks near initialization. It does not describe finite-width feature learning, and it does not hold for every architecture, parameterization, lo...

StatusChecked against the cited passages.

Checked the papers' LaTeX source: Jacot et al. support the NTK as the kernel of parameter gradients, which becomes deterministic and constant in the infinite-width limit and turns least-squares gradient flow into kernel gradient descent; Lee et al. support the first-order Taylor (linearized) view near initialization for sufficiently wide networks. The page's equations, code and demo show the kernel ODE with the gradients held fixed, and only in that regime.

Explain it without the pageWrite your own explanation before asking for feedback. Your draft and any hints stay while this page is open.

Practice · Neural Tangent Kernel (NTK) & Infinite-Width Limits

Try the idea in your own words

In the infinite-width limit under the NTK parameterization, squared-loss training by gradient flow or with a sufficiently small learning rate follows the network's linearization at initialization. When the training kernel is invertible, the converged predictions add a kernel-regression fit of the initial errors to the initial predictions. The notebook contrasts this fixed-kernel regime with feature learning.

Concept · Selected for practice

Neural Tangent Kernel (NTK) & Infinite-Width Limits

What it rests on: Sources: Neural Tangent Kernel: Convergence and Generalization in Neural Networks; Wide Neural Networks of Any Depth Evolve as Linear Models Under...

Context and links

Scaling

Choose a task

Explain the mechanism

For Neural Tangent Kernel (NTK) & Infinite-Width Limits: What is the smallest example of Neural Tangent Kernel (NTK) & Infinite-Width Limits you can work through by hand, and what does it show? Explain your answer, including what changes, why, and which assumption matters.

No answer yet

A rough first thought is enough. Your draft stays when you change tasks.

Your draft stays on this page and is cleared when you leave or reload.

A little help · Explain

Open one hint at a time. These are suggestions, not your answer or a grade.

0 of 3 hints shown for this question.

    Clearing your answer keeps this help record. Outside help is your own report; this page cannot check it.

    Where am I stuck? (optional)
    Your own description, not an automatic diagnosis

    Choose one, or leave this unspecified. Select it again to clear it.

    Take your draft to a feedback conversation

    No AI feedback runs here. You can copy a prompt to use elsewhere; nothing is sent automatically. Review the text before sharing, and leave out private information.

    Write an attempt before copying a feedback prompt.

    A draft, or an AI reply to it, is not a test of what you have learned. To check that, try a different case later without help.

    Ask about this pagePick one item on this page and copy a prompt about it into an AI assistant you already use. Notes you write here stay in this browser.
    Ask about this pageClose
    ConceptNeural Tangent Kernel (NTK) & Infinite-Width LimitsSources: Neural Tangent Kernel: Convergence and Generalization in Neural Networks; Wide Neural Networks of Any Depth Evolve as Linear Models Under...rs = np.random.RandomState(0)

    Your question

    Choose what your question is about

    Pick the idea, equation, source, code, claim, misconception or demo state you want to ask about. The prompt you copy includes it, so the answer can stay on that item.
    Next local actionNo local draft saved yet

    Open the draft below to save one note and next action in this browser.

    conceptScaling

    Neural Tangent Kernel (NTK) & Infinite-Width Limits

    Starting question

    What is the smallest example of Neural Tangent Kernel (NTK) & Infinite-Width Limits you can work through by hand, and what does it show?

    SourcesCheck the 3 cited sources listed under Sources.Notes can be saved for this item
    Points of view on this item

    Fixed prompts generated from this item. They are not comments from people or independent reviews.

    Learner’s first stepAsk what would make "Neural Tangent Kernel (NTK) & Infinite-Width Limits" feel predictable rather than familiar.
    Assumption

    The 3 cited sources must support this exact item, not just the surrounding topic.

    Source-checking summary

    Connect the definition to one equation, piece of code or demo before widening the discussion.

    Proposed experiment

    Change one input, then check that the same relationship holds in the mathematics, the code and the demo.

    Next action

    You can state the mechanism in your own words

    Evidence4 checks
    ObservationChecking for a saved observation
    ActionReady for one action
    PromptLearner prompt ready to copy
    Go to this item
    01ObservationChecking what this browser has saved
    02EvidenceChecking for a saved observation
    03SourcesCheck the 3 cited sources listed under Sources.
    04Next stepSave one next action
    Local action draftNo local draft saved yetOpen when you are ready to write one next action
    Local action draft

    This draft stays in this browser and is attached to this item only.

    No local draft saved.
    Evidence to inspect
    • What the 3 cited sources say about this exact item
    • The definition, its prerequisites, and a contrasting concept
    • The equation or code that makes the concept concrete
    • One demo state that shows what stays the same as the inputs change
    What would resolve this
    • You can state the mechanism in your own words
    • You can name the prerequisite that would clear up the confusion
    • You can predict how the result changes when one input changes
    Reference idconcept/concept-notebook/scaling/ntk concept:scaling/ntk