Computation Graphs

In this first example, every node is scalar. Let $x,y\in\mathbb R$, and draw edges from inputs to the values that depend on them. The graph for one forward pass is a directed acyclic graph: later nodes depend on earlier nodes, not the other way around.

Consider the scalar computation

$$a = xy,\qquad b = \sin(a),\qquad c = a + b.$$

The graph has inputs $x,y$, intermediate nodes $a,b$, and output $c$. The forward pass stores each node value. The important detail is reuse: $a$ is stored once, but two later uses depend on it. The backward pass asks how a small change in each node would affect the output.

For any node $v$, write

$$\bar v = \frac{\partial c}{\partial v}.$$

The output seed is $\bar c=1$. The final add node $c=a+b$ sends one unit of sensitivity to both inputs:

$$\bar a \mathrel{+}= \bar c\frac{\partial c}{\partial a}\Big|_{\text{direct}} = 1,\qquad \bar b \mathrel{+}= \bar c\frac{\partial c}{\partial b} = 1.$$

The sine branch also sends a local contribution into the same stored node $a$. The important invariant is local: a node's cotangent is the sum of contributions from each immediate downstream use. The demo hides the sine-branch contribution until you predict whether it will lower, preserve, or raise the direct baseline.

The backward pass does not expand every symbolic path; each downstream node has already summarized everything beyond it.

For vector or tensor nodes, the same idea uses Jacobians or local vector-Jacobian product rules. With column-vector cotangents, if $v_j=f(v_i)$ and $J_{ji}=\partial v_j/\partial v_i$, the reverse update is

$$\bar v_i \mathrel{+}= J_{ji}^{\mathsf T}\bar v_j.$$

Reverse-mode autodiff is the automated version of this bookkeeping: save forward values, then run local backward rules in reverse topological order.