Taylor Expansion and Local Approximation

Let $f:\mathbb{R}\to\mathbb{R}$ be differentiable near $x_0$, and define the displacement

$$h = x - x_0.$$

The first-order Taylor model is

$$T_1(x_0+h)=f(x_0)+f'(x_0)h.$$

This is the tangent-line prediction: keep the current value and add slope times displacement.

If $f$ has a second derivative near $x_0$, the second-order Taylor model is

$$T_2(x_0+h)=f(x_0)+f'(x_0)h+\frac{1}{2}f''(x_0)h^2.$$

The new term is the curvature correction. The factor $1/2$ is not decoration: it is the coefficient that makes the quadratic model match the second derivative at the anchor.

For a smooth function with higher derivatives, the degree-$n$ Taylor polynomial is

$$T_n(x_0+h)=\sum_{k=0}^{n}\frac{f^{(k)}(x_0)}{k!}h^k.$$

The local caveat is the point. A Taylor polynomial is exact for a polynomial of low enough degree, and often excellent near $x_0$, but the remainder can dominate far from the anchor.

The multivariable version is the one optimization uses. For $f:\mathbb{R}^d\to\mathbb{R}$, current point $x_0\in\mathbb{R}^d$, and small displacement $\delta\in\mathbb{R}^d$,

$$f(x_0+\delta)\approx f(x_0)+\nabla f(x_0)^\mathsf{T}\delta +\frac{1}{2}\delta^\mathsf{T}H(x_0)\delta,$$

where $\nabla f(x_0)$ is the gradient and $H(x_0)$ is the Hessian matrix.

Gradient descent comes from the first-order term. If $\delta=-\eta\nabla f(x_0)$, then

$$\nabla f(x_0)^\mathsf{T}\delta =-\eta\|\nabla f(x_0)\|_2^2,$$

which predicts a local decrease. The second-order term warns that curvature can shrink, improve, or overturn that prediction when the step is too large.