Reverse-Mode Automatic Differentiation

Let a differentiable computation graph produce a scalar output $L$ from intermediate variables $v_1,\dots,v_n$. Reverse mode stores, for each node, an adjoint or cotangent

$$\bar v_i = \frac{\partial L}{\partial v_i}.$$

Before the reverse sweep, initialize every non-output cotangent register to zero. Then seed the scalar output with

$$\bar L = \frac{\partial L}{\partial L} = 1.$$

For scalar nodes and a local operation $v_j = f(v_i)$, the chain rule sends sensitivity backward. When the operation producing $v_j$ is processed, $\bar v_j$ already contains all downstream contributions:

$$\bar v_i \mathrel{+}= \bar v_j \frac{\partial v_j}{\partial v_i}.$$

For an operation with multiple inputs, such as $v_j=f(u,w)$, each input receives its own local derivative:

$$\bar u \mathrel{+}= \bar v_j \frac{\partial v_j}{\partial u},\qquad \bar w \mathrel{+}= \bar v_j \frac{\partial v_j}{\partial w}.$$

The plus-equals matters. If a value is reused by several later nodes, all downstream paths contribute to its total sensitivity. Reverse mode is therefore not symbolic simplification; it is graph-local accumulation of vector-Jacobian products.

For vector nodes, choose column-vector cotangents. If $v_i\in\mathbb{R}^n$, $v_j=f(v_i)\in\mathbb{R}^m$, $\bar v_i\in\mathbb R^n$, $\bar v_j\in\mathbb R^m$, and

$$J_{ji}=\frac{\partial v_j}{\partial v_i}\in\mathbb{R}^{m\times n},$$

then the reverse update is

$$\bar v_i \mathrel{+}= J_{ji}^{\mathsf T}\bar v_j.$$

This is the direction contrast:

$$\text{JVP: }\dot v_j=J_{ji}\dot v_i,\qquad \text{VJP: }\bar v_i=J_{ji}^{\mathsf T}\bar v_j.$$

For a scalar loss $L:\mathbb R^p\to\mathbb R$, one forward evaluation records the needed values, and one reverse sweep gives the full gradient

$$\nabla_\theta L = \left(\frac{\partial L}{\partial \theta_1},\dots,\frac{\partial L}{\partial \theta_p}\right)^{\mathsf T}$$

assuming primitive backward rules are available. The cost is memory for saved forward values on a tape, or extra compute if some values are recomputed. An autodiff engine automates this bookkeeping by recording primitive operations during the forward pass and executing their local backward rules in reverse topological order.

These equations assume the recorded primitives are differentiable at the saved forward values. For nonsmooth primitives, an implementation must choose a convention, use a subgradient, or report that the derivative is undefined. If the program has control flow, reverse mode differentiates the branch that actually ran.