Reverse-Mode Automatic Differentiation

Let a differentiable computation graph produce a scalar output $L$ from intermediate variables $v_1,\dots,v_n$. Reverse mode stores, for each node, an adjoint or cotangent

Before the reverse sweep, initialize every non-output cotangent register to zero. Then seed the scalar output with

For scalar nodes and a local operation $v_j = f(v_i)$, the chain rule sends sensitivity backward. When the operation producing $v_j$ is processed, $\bar v_j$ already contains all downstream contributions:

For an operation with multiple inputs, such as $v_j=f(u,w)$, each input receives its own local derivative:

The plus-equals matters. If a value is reused by several later nodes, all downstream paths contribute to its total sensitivity. Reverse mode is therefore not symbolic simplification; it is graph-local accumulation of vector-Jacobian products (VJPs): each step multiplies a vector of sensitivities by the transpose of one local Jacobian.

For vector nodes, choose column-vector cotangents. If $v_i\in\mathbb{R}^n$, $v_j=f(v_i)\in\mathbb{R}^m$, $\bar v_i\in\mathbb R^n$, $\bar v_j\in\mathbb R^m$, and

then the reverse update is

Forward mode runs the other way. It carries a tangent $\dot v$, the change in each node along one chosen input direction, forward with Jacobian-vector products (JVPs). The two directions side by side:

For a scalar loss $L:\mathbb R^p\to\mathbb R$, one forward evaluation records the needed values, and one reverse sweep gives the full gradient

assuming primitive backward rules are available. The time is a small constant multiple of one evaluation of $L$, whatever $p$ is: Baydin et al. (2018), citing Griewank and Walther (2008), give a constant below 6 and typically 2 to 3. The price is memory for saved forward values on a tape, or extra compute if some values are recomputed. An autodiff engine automates this bookkeeping by recording primitive operations during the forward pass and executing their local backward rules in reverse topological order.

These equations assume the recorded primitives are differentiable at the saved forward values. For nonsmooth primitives, an implementation must choose a convention, use a subgradient, or report that the derivative is undefined. If the program has control flow, reverse mode differentiates the branch that actually ran.