Continuous Function

Calculus

Jacobians, Hessians, VJPs, and JVPs

Jacobians are local linear maps; JVPs push perturbations forward, VJPs pull sensitivities back, and Hessian-vector products expose curvature without building every entry.

status: reviewimportance: criticaldifficulty 4/5math: graduateread: 15mlive demo
Back to CalculusNext: Reverse-Mode Automatic Differentiation

Concept Structure

Jacobians, Hessians, VJPs, and JVPs

01Intuition

Start with the picture, metaphor, or geometric mechanism.

02Math

Make the objects explicit and connect them with notation.

03Code

Mirror the equations with runnable implementation details.

04Interactive Demo

Manipulate the mechanism and watch the idea respond.

3prerequisites
5next concepts
3related links

Learner Contract

What this page should let you do.

You are here becauseJacobians are local linear maps; JVPs push perturbations forward, VJPs pull sensitivities back, and Hessian-vector products expose curvature without building every entry.

This Calculus concept is the current idea: keep the same invariant visible across Intuition, Math, Code, Interactive Demo.

By the end4/4 sections ready | runnable code expected | live demo

Explain the mechanism, trace the main notation, and test one prediction in the live demo.

Do this firstIntuition

Read the intuition before the notation; the math should name a mechanism you already felt.

Test the linkManipulate one control and predict the visible change.Then continue to Reverse-Mode Automatic Differentiation

Claim/source review status

Substantive review recorded

1/1 claims have bounded review metadata; still check caveats and source scope.Metadata-derived; review may be AI-assisted. Not a human certification.
Claims1/1 reviewed
Sources5 cited
Codeattached
Demolive
Reviewed2026-07-02
Updatedpage 2026-07-02

Learning item flow

4/4 sections readyAsk about thisResearch room
ConceptJacobians, Hessians, VJPs, and JVPsCalculusEquationJacobians, Hessians, VJPs, and JVPs equation 1Exact equationCodeJacobians, Hessians, VJPs, and JVPs runnable code 1Exact runnable codeDemoJacobians, Hessians, VJPs, and JVPs interactive demoInteractive demoClaimFor smooth finite-dimensional functions, JVPs push input tangents for...Exact claim checkSourceMathematics for Machine LearningExact source
ConceptJacobians, Hessians, VJPs, and JVPsCalculus
5 sources attachedLocal snapshot ready
concept:calculus/jacobians-hessians-vjp-jvp
CodenearbyPredictbefore revealRoomitem handoff
01IntuitionStart with the picture, metaphor, or geometric mechanism.02MathMake the objects explicit and connect them with notation.03CodeMirror the equations with runnable implementation details.04Interactive DemoManipulate the mechanism and watch the idea respond.
01

01

Intuition

Build the mental picture first so the rest of the page has something to attach to.

Section prompt

A derivative in one dimension is a slope. In many dimensions, the derivative is a local linear map.

Near an anchor point x0x_0, a differentiable function f:Rn→Rmf:\mathbb{R}^n\to\mathbb{R}^m behaves like

f(x0+δ)≈f(x0)+J(x0)δ.f(x_0+\delta)\approx f(x_0)+J(x_0)\delta.

The Jacobian J(x0)J(x_0) is the matrix for that local linear map. But deep-learning systems often do not want the whole matrix. They want the matrix acting on a vector.

There are two directions to keep separate.

A JVP asks: "if the input moves a little in direction vv, how does the output move?"

A VJP asks: "if a downstream objective cares about the output through covector uu, how should that sensitivity be pulled back to the input?"

That shape discipline is the whole concept:

  • JvJv moves tangent information forward.
  • JTuJ^\mathsf{T}u moves cotangent information backward.
  • HvHv moves curvature information through the Hessian of a scalar loss.

The full matrix is sometimes useful for inspection. The products are what make modern autodiff practical.

02

02

Math

Translate the story into symbols, assumptions, and a derivation you can inspect.

Section prompt
Equation 1f:Rn→Rm,x↦f(x).f:\mathbb{R}^n\to\mathbb{R}^m,\qquad x\mapsto f(x).Equation 2J(x0)=[∂f1∂x1⋯∂f1∂xn⋮⋱⋮∂fm∂x1⋯∂fm∂xn]∈Rm×n.J(x_0)= \begin{bmatrix} \frac{\partial f_1}{\partial x_1} & \cdots & \frac{\partial f_1}{\partial x_n}\\ \vdots & \ddots & \vdots\\ \frac{\partial f_m}{\partial x_1} & \cdots & \frac{\partial f_m}{\partial x_n} \end{bmatrix} \in\mathbb{R}^{m\times n}.

Let

f:Rn→Rm,x↦f(x).f:\mathbb{R}^n\to\mathbb{R}^m,\qquad x\mapsto f(x).

At an anchor x0∈Rnx_0\in\mathbb{R}^n, the Jacobian is

J(x0)=[∂f1∂x1⋯∂f1∂xn⋮⋱⋮∂fm∂x1⋯∂fm∂xn]∈Rm×n.J(x_0)= \begin{bmatrix} \frac{\partial f_1}{\partial x_1} & \cdots & \frac{\partial f_1}{\partial x_n}\\ \vdots & \ddots & \vdots\\ \frac{\partial f_m}{\partial x_1} & \cdots & \frac{\partial f_m}{\partial x_n} \end{bmatrix} \in\mathbb{R}^{m\times n}.

The first-order Taylor approximation says

f(x0+δ)≈f(x0)+J(x0)δ.f(x_0+\delta)\approx f(x_0)+J(x_0)\delta.

If v∈Rnv\in\mathbb{R}^n is an input tangent direction, the Jacobian-vector product is

JVP⁡(x0,v)=J(x0)v∈Rm.\operatorname{JVP}(x_0,v)=J(x_0)v\in\mathbb{R}^m.

It answers a forward perturbation question: one input-space direction becomes an output-space direction.

If u∈Rmu\in\mathbb{R}^m is an output cotangent, the vector-Jacobian product is

VJP⁡(x0,u)=J(x0)Tu∈Rn.\operatorname{VJP}(x_0,u)=J(x_0)^\mathsf{T}u\in\mathbb{R}^n.

It answers a reverse sensitivity question: one output-space scoring vector becomes an input-space sensitivity.

For a scalar loss L:Rn→RL:\mathbb{R}^n\to\mathbb{R}, the gradient is input-shaped:

∇L(x0)∈Rn.\nabla L(x_0)\in\mathbb{R}^n.

The Hessian collects second derivatives:

H(x0)=∇2L(x0)∈Rn×n.H(x_0)=\nabla^2 L(x_0)\in\mathbb{R}^{n\times n}.

A Hessian-vector product

H(x0)p∈RnH(x_0)p\in\mathbb{R}^n

answers a curvature question: how does the gradient change if the input moves in direction pp?

For neural networks with millions or billions of parameters, materializing full Jacobians or Hessians is usually the wrong first mental model. Autodiff systems expose products because products preserve the useful local action without storing every entry.

03

03

Code

Keep the implementation aligned with the notation so the algorithm is legible.

Section prompt
Runnable code 1import numpy as np def f(x): a, b = x return np.array([a*a + 0.5*b, np.sin(a) + a*b, a - b*b]...python
import numpy as np

def f(x):
    a, b = x
    return np.array([a*a + 0.5*b,
                     np.sin(a) + a*b,
                     a - b*b])

def jacobian(x):
    a, b = x
    return np.array([[2*a, 0.5],
                     [np.cos(a) + b, a],
                     [1.0, -2*b]])

def hessian_loss(x):
    a, _ = x
    return np.array([[6*a, 0.5],
                     [0.5, 2.0]])

x0 = np.array([0.8, -0.4])
v = np.array([0.4, -0.7])      # input tangent, shape (2,)
u = np.array([0.6, -0.8, 0.5]) # output cotangent, shape (3,)
p = np.array([0.25, -0.6])     # curvature probe, shape (2,)

J = jacobian(x0)
print("f(x0):", np.round(f(x0), 3))
print("J shape:", J.shape)
print("Jv:", np.round(J @ v, 3))
print("J^T u:", np.round(J.T @ u, 3))
print("H p:", np.round(hessian_loss(x0) @ p, 3))

Notice the shapes. Here JJ is 3×23\times 2. The JVP has output shape (3,). The VJP returns to input shape (2,). The Hessian-vector product for a scalar loss also has input shape (2,).

04

04

Interactive Demo

Use direct manipulation to connect the explanation to a moving system.

Section prompt

Use the lab below as a shape-first autodiff witness. You can inspect the anchor, tangent vector, output cotangent, and the fact that f:R2→R3f:\mathbb{R}^2\to\mathbb{R}^3.

Before revealing numbers, answer this question: an output cotangent u∈R3u\in\mathbb{R}^3 arrives from a downstream score. Which operation sends that sensitivity back to the two input coordinates?

Then reveal the local Jacobian, the JVP, the VJP, and a Hessian-vector mini-check. Try the presets after the default case. The values change, but the shape contract should not.

Live Concept Demo

Explore Jacobians, Hessians, VJPs, and JVPs

The stage is code-native and interactive. Use it to test the explanation against the mechanism.

difficulty 4/5graduatecode-aligned
Demo Prediction Checkpoint

Manipulate one control and predict the visible change.

Commit to what Jacobians, Hessians, VJPs, and JVPs should make visible before reading the result.

After The First Pass

Turn the concept into an inspected object.

Once the invariant is visible in the intuition, math, code, and demo, use these panels to inspect the mechanism visually, check source support, practice the idea, and attach a grounded research question.

Mechanism Storyboard

See the idea move before the page explains it

Jacobians are local linear maps; JVPs push perturbations forward, VJPs pull sensitivities back, and Hessian-vector products expose curvature without building every entry.

Prediction open01 / Intuition
Prediction lens

Start with the picture, metaphor, or geometric mechanism.

Commit first

Before reading further, choose the kind of change Jacobians, Hessians, VJPs, and JVPs should make visible.

Visual Inquiry

Make the image answer a mathematical question

Jacobians are local linear maps; JVPs push perturbations forward, VJPs pull sensitivities back, and Hessian-vector products expose curvature without building every entry.

4/4 stages readyLive demo connected
Prediction

Which visible object should carry the first intuition?

Commit first

Pick the cue that should make Jacobians, Hessians, VJPs, and JVPs easier to reason about before the page gives the answer.

Source Grounding

Canonical references for the mechanism on this page.

book · 2020Mathematics for Machine LearningDeisenroth, Faisal, and Ong

Supports Jacobian matrix shape, Hessian matrix/tensor shape, and the multivariate Taylor/local-linearization bridge.

Open source
paper · 2018Automatic Differentiation in Machine Learning: a SurveyBaydin, Pearlmutter, Radul, and Siskind

Supports forward-mode matrix-free Jacobian-vector products, reverse-mode transposed Jacobian-vector products, and Hessian-vector products.

Open source
documentation · 2026JAX Autodiff CookbookJAX Authors

Supports JVP/VJP math, jacfwd/jacrev tradeoffs, and Hessian-vector product recipes.

Open source
documentation · 2026PyTorch autograd.functional jvp/vjp/jacobian/hessianPyTorch Contributors

Supports practical API shape contracts: jvp vector same size as input and result shaped like output; vjp vector same size as output and result shaped like input; hessian for scalar functions.

Open source
book · 2026Dive into Deep Learning: Automatic DifferentiationZhang, Lipton, Li, and Smola

Supports the framework behavior that backpropagation usually returns vector-Jacobian products rather than materializing full Jacobians.

Open source

Claim Review

Jacobians are local linear maps; JVPs push perturbations forward, VJPs pull sensitivities back, and Hessian-vector products expose curvature without building every entry.

Status1 substantive review recorded

Claims without a substantive review badge still need exact source-support review.

Sources5 references

deisenroth-2020-mml-jacobian-hessian, baydin-2018-ad-survey-jvp-vjp, jax-2026-autodiff-cookbook, pytorch-2026-autograd-functional, d2l-2026-autograd

Local checks4 local checks

Use equations, runnable code, and demos to check whether the source support is operational.

Substantively reviewedFor smooth finite-dimensional functions, JVPs push input tangents forward, VJPs pull output cotangents back, and HVPs expose scalar-loss curvature action without dense derivative matrices.Claim metadata: source checked

The sources agree on the shape story: Jacobians collect first-order partial derivatives, JVPs push input tangents forward, VJPs apply the transposed Jacobian to output cotangents, and Hessian-vector products are a practical curvature primitive.

Sources: Mathematics for Machine Learning, Automatic Differentiation in Machine Learning: a Survey, JAX Autodiff Cookbook, PyTorch autograd.functional jvp/vjp/jacobian/hessian, Dive into Deep Learning: Automatic DifferentiationThis page teaches finite-dimensional real-valued arrays and local smooth functions. It does not cover complex differentiation conventions, nonsmooth primitives, sparse Jacobian compression, custom derivative rules, checkpointing policy, or full framework edge cases.A bounded review summary is present; still check caveats and exact reference scope.

MML, Baydin, JAX, PyTorch, and D2L support the Jacobian/Hessian shape story and product-first autodiff framing; GPT Pro critique remains pending.

Reviewer: codex-local-source-review; reviewed 2026-07-02
Equation 1Equation 2Runnable code 1Interactive demo

Source support candidates

book 2020Mathematics for Machine Learning

Supports Jacobian matrix shape, Hessian matrix/tensor shape, and the multivariate Taylor/local-linearization bridge.

paper 2018Automatic Differentiation in Machine Learning: a Survey

Supports forward-mode matrix-free Jacobian-vector products, reverse-mode transposed Jacobian-vector products, and Hessian-vector products.

documentation 2026JAX Autodiff Cookbook

Supports JVP/VJP math, jacfwd/jacrev tradeoffs, and Hessian-vector product recipes.

documentation 2026PyTorch autograd.functional jvp/vjp/jacobian/hessian

Supports practical API shape contracts: jvp vector same size as input and result shaped like output; vjp vector same size as output and result shaped like input; hessian for scalar functions.

Mechanism checks

Equation 1
f:Rn→Rm,x↦f(x).f:\mathbb{R}^n\to\mathbb{R}^m,\qquad x\mapsto f(x).
Equation 2
J(x0)=[∂f1∂x1⋯∂f1∂xn⋮⋱⋮∂fm∂x1⋯∂fm∂xn]∈Rm×n.J(x_0)= \begin{bmatrix} \frac{\partial f_1}{\partial x_1} & \cdots & \frac{\partial f_1}{\partial x_n}\\ \vdots & \ddots & \vdots\\ \frac{\partial f_m}{\partial x_1} & \cdots & \frac{\partial f_m}{\partial x_n} \end{bmatrix} \in\mathbb{R}^{m\times n}.
Runnable code 1import numpy as np def f(x): a, b = x return np.array([a*a + 0.5*b, np.sin(a) + a*b, a - b*b]...Demo stateLive mechanism probe

Practice Loop

Try the idea before it explains itself

Jacobians are local linear maps; JVPs push perturbations forward, VJPs pull sensitivities back, and Hessian-vector products expose curvature without building every entry.

Readiness0/3 checks ready
Predict

Before touching the demo, predict one visible change that should happen in Jacobians, Hessians, VJPs, and JVPs.

Hint 1

Reveal when your model needs a nudge.

Hint 2

Reveal when your model needs a nudge.

Hint 3

Reveal when your model needs a nudge.

Grounded research drawerClose
ConceptJacobians, Hessians, VJPs, and JVPsCalculus
Runnable code comparisonJacobians, Hessians, VJPs, and JVPs runnable code 1a, b = xPrediction before revealJacobians, Hessians, VJPs, and JVPs interactive demoManipulate one control and predict the visible change.
Grounded room questionWhat is the smallest example that makes Jacobians, Hessians, VJPs, and JVPs click without losing the math?Local snapshot ready

Research Room

Attach the question to a claim, equation, code, or demo

Pick the concept, equation, source, runnable code, claim, misconception, or demo state before asking for help. The handoff keeps that page item in context.
Next local actionNo local draft saved yet

Open the draft below to save one note and next action in this browser.

conceptCalculus

Jacobians, Hessians, VJPs, and JVPs

Attached question

What is the smallest example that makes Jacobians, Hessians, VJPs, and JVPs click without losing the math?

Local action draftNo local draft saved yetExpand only when ready to capture one local next action
Local action draft

This draft stays in this browser, attached to the selected learning item.

No local draft saved.
Evidence to inspect
  • References to inspect: attached references on this page.
  • Definition, prerequisite, and contrast concept links
  • The equation or runnable code that makes the concept operational
  • One demo state that shows the invariant instead of a slogan
What would resolve this
  • The learner can state the mechanism in their own words
  • The learner can name the prerequisite that would repair confusion
  • The learner can predict how the mechanism changes under one perturbation
Grounded AI handoff

I am working in Continuous Function's research reading room. Object: concept - Jacobians, Hessians, VJPs, and JVPs Selected item key: recorded for copy. Context: Calculus Page anchor: recorded for copy. Open question: What is the smallest example that makes Jacobians, Hessians, VJPs, and JVPs click without losing the math? Evidence to inspect: - References to inspect: attached references on this page. - Definition, prerequisite, and contrast concept links - The equation or runnable code that makes the concept operational - One demo state that shows the invariant instead of a slogan What would resolve this: - The learner can state the mechanism in their own words - The learner can name the prerequisite that would repair confusion - The learner can predict how the mechanism changes under one perturbation Answer as a careful research tutor: stay source-grounded, separate verified evidence from assumptions, name the relevant math objects, and end with one next action.

View it in context
concept/concept-notebook/calculus/jacobians-hessians-vjp-jvp concept:calculus/jacobians-hessians-vjp-jvp