Derivatives

How can a curve have a slope at a single point, when a slope needs two points?

The idea

Take a second point a step $h$ away, on either side, and measure the slope of the line through both points, the secant. If that slope approaches the same finite number from both sides as $h$ shrinks to zero, that number is the derivative: the slope of the tangent line at the point.

Builds on Functions.

Functions has no page here yet. Read OpenStax, Calculus Volume 1, Section 1.1: Review of Functions.

Ready? For f(x) = (x³ − 3x)/2, what is f(2)?

Show the answer

f(2) = (8 − 6)/2 = 1.

The picture

Slope is rise over run, and that takes two points. So take two: the point on the curve, and a second one a step $h$ to its right. The line through them is the secant. Its slope, $(f(x+h) - f(x))/h$ , is the curve's average steepness over that step, and a long step averages over a whole stretch of curve.

Shrink $h$ and the secant pivots about the point until it settles on a single line, the tangent, which grazes the curve there. The tangent's slope is the derivative, $f'(x)$ : the steepness at the point itself, not across a stretch.

The lens magnifies the neighbourhood of the point, more as $h$ shrinks. Magnify a smooth curve enough and it looks straight; that straight line is the tangent. The chart underneath draws $f'(x)$ at every $x$ at once: above zero where the curve rises, below zero where it falls, and exactly zero where the tangent lies flat. A corner, like the one in $|x|$ at zero, never straightens however far you zoom.

In machine learning the curve is a model's loss and $x$ is one of its parameters. The derivative says which way to nudge that parameter to lower the loss, and how strongly the loss responds.

Zoom in until the curve is a straight line.

Drag the point along the curve. The secant runs to a second point a step h away; shrink h and it swings onto the tangent. The tangent’s slope is the derivative — and the lower chart draws it at every x.

The curve with its secant and tangent, and below it the derivative curve

Drag anywhere to move the point (or focus it and use the arrow keys). Drag the orange dot to change h.

Zoom lens×2

The curve still bends away from the tangent. Keep shrinking h.

Step h1

tangent slope f′(x): −1.125
secant slope: 0.125
secant misses by: 1.25

Show the left side too

At x = 0.5 the curve is falling: the tangent’s slope is f′(x) = −1.125. Your secant over h = 1 has slope 0.125, 1.25 off. Shrink h and it swings onto the tangent.

On the hill and valley, drag the point to where the tangent lies perfectly flat.

One input and three hand-picked curves: with many parameters, each gets a slope like this, and together they form the gradient.

The mathematics

Definition (derivative). The derivative of $f$ at $x$ is

f'(x) = \lim_{h \to 0} \frac{f(x+h) - f(x)}{h}.

The fraction

\frac{f(x+h) - f(x)}{h}

is a secant slope: the average rate of change over the step. When the limit exists, from both sides and to the same finite number, it is the tangent slope. Steps from the right alone do not establish it.

Once you know $f'(x_0)$ , the tangent line at $x_0$ is

y = f(x_0) + f'(x_0)(x - x_0).

For a positive step $h$ , the left, right and central quotients at the same point are

\begin{aligned} D_-(h) &= \frac{f(x)-f(x-h)}{h}, \\ D_+(h) &= \frac{f(x+h)-f(x)}{h}, \\ D_c(h) &= \frac{f(x+h)-f(x-h)}{2h} = \frac{D_-(h)+D_+(h)}{2}. \end{aligned}

The derivative exists when $D_-(h)$ and $D_+(h)$ tend to the same finite limit as $h$ shrinks. For a smooth function, the one-sided quotients miss $f'(x)$ by an amount proportional to $h$ , while the central quotient's leading errors cancel and it misses by an amount proportional to $h^2$ . The central quotient can mislead at a corner: for $|x|$ at zero, $D_+ = 1$ and $D_- = -1$ at every $h$ , yet $D_c = 0$ at every $h$ .

Try it

First a worked example, then one to finish, then problems on your own. Check your answer before you look at the solution. Answers typed here are not kept when you leave.

1 · Worked example
The slope at x = 0.5, from the definition
For $f(x) = (x^3 - 3x)/2$ , find $f'(0.5)$ as the limit of the secant's slope.
1. Rise. Expand $(x+h)^3 = x^3 + 3x^2h + 3xh^2 + h^3$ , so $f(x+h) - f(x) = \tfrac{1}{2}(3x^2h + 3xh^2 + h^3 - 3h)$ .
Say the next step to yourself first.
2 · Finish it
Average both sides
Instead of one secant to the right, take the line from $x - h$ to $x + h$ , which is the average of the left and right secants. With $x = 0.5$ and $h = 0.1$ , finish the estimate of $f'(0.5)$ .
1. Right point. $f(0.6) = (0.216 - 1.8)/2 = -0.792$ .
2. Left point. $f(0.4) = (0.064 - 1.2)/2 = -0.568$ .
3. Rise across both sides. $f(0.6) - f(0.4) = -0.224$ .
4. Your turn.
Your answer
3 · On your own
A ten-times-smaller step
At $x = 0.5$ with $h = 0.1$ , the one-sided secant misses $f'(0.5)$ by $0.08$ and the two-sided average misses by $0.005$ . Shrink $h$ to $0.01$ . Roughly how far off is each now?
Choose oneUnchanged: 0.08 and 0.005Both ten times smaller: about 0.008 and 0.0005About 0.008 one-sided, and about 0.00005 for the averageBoth a hundred times smaller: about 0.0008 and 0.00005
4 · On your own
Which way is downhill?
Suppose $f(x) = (x^3 - 3x)/2$ is a model's loss and $x$ its one parameter, currently $0.5$ , where $f'(0.5) = -1.125$ . Training nudges $x$ a small amount at a time to lower the loss. Which way should it move, and where do many small nudges end up?
Choose oneLeft, toward the hill at x = −1Right, and they stop at the valley, x = 1Right, and they keep going foreverNeither way: a negative slope means the loss is already at its lowest

The code

The definition, run with shrinking steps on $\sin x$ at $x = 1.2$ , where the true slope is $\cos 1.2 \approx 0.362$ .

import numpy as np

f = lambda x: np.sin(x)
df_true = lambda x: np.cos(x)

x0 = 1.2
for h in [1.0, 0.3, 0.1, 0.03, 0.01]:
    df_est = (f(x0 + h) - f(x0)) / h
    print("h=", h, "secant=", df_est, "true=", df_true(x0))

The secant's slope climbs from $-0.124$ at $h = 1$ to $0.358$ at $h = 0.01$ . Each ten-fold shrink of $h$ cuts the gap about ten-fold: $0.047$ at $h = 0.1$ , $0.0047$ at $h = 0.01$ .

Keep it

Zoom in far enough on a smooth curve at one point. What do you see, and what is its slope?
Answer from memory, then check
A straight line: the tangent. Its slope is the derivative $f'(x)$ , the steepness of the curve at that point.
You shrink $h$ ten times. How does the miss change for a one-sided secant, and for the average of both sides?
Answer from memory, then check
On a smooth curve the one-sided miss shrinks about ten times (in step with $h$ ). The average's miss shrinks about a hundred times (like $h^2$ ), because the leading errors of the two sides cancel.
True or false: wherever $f'(x) = 0$ , the curve has a peak or a valley.
Answer from memory, then check
False. Peaks and valleys of a smooth curve do have $f'(x) = 0$ , but $x^3$ is flat at $0$ and keeps rising through it. A zero slope says where to look, not what you will find.
Why does $|x|$ have no derivative at $x = 0$ ?
Answer from memory, then check
The right secant has slope $1$ and the left has slope $-1$ at every $h$ . Zooming in never makes them agree, so no single tangent fits. Their average, $0$ , is steady but describes neither side.
Why does training a model need derivatives at all?
Answer from memory, then check
The derivative of the loss with respect to a parameter says which way, and how strongly, the loss changes when that parameter is nudged. A small step against it lowers the loss; without it, every nudge would be a blind guess.

Recalling an answer from memory makes it last longer than rereading it (Roediger and Karpicke, 2006). Add these 5 cards and they come back tomorrow, then after gaps of 3, 7, 16, 35 and 75 days while you still remember them. They stay in this browser only.

The edge

Everything here assumed a smooth curve, but the networks trained in practice are full of corners. ReLU, $\max(0, x)$ , has exactly the corner of $|x|$ at zero, where no derivative exists. Software simply picks a slope there, usually 0, and training works anyway. For convex functions the choice is justified by subgradients: any slope of a line that touches the function from below without crossing it. For deep networks the theory is still being built. At some points, automatic differentiation of such programs can return a value that is neither the derivative nor any subgradient. Bolte & Pauwels (2020) show that such points form a negligible set and give a framework for reasoning about them, but which slopes training actually needs there is not settled. In practice the choice still matters: in the 32- and 16-bit arithmetic used for training, the slope chosen for ReLU at zero can measurably change the result (Bertoin et al., 2021).

Next thoughtThe derivative says which way is downhill. How big a step should you take, and what goes wrong when the step is too big?Gradient Descent →

The full instrument: predict the hidden tangent, and compare both sides step by step on |x| and |x − 1|

The picture

The mathematics

Try it

The slope at x = 0.5, from the definition

Average both sides

A ten-times-smaller step

Which way is downhill?

The code

Keep it

The edge