Convex Sets, Convex Functions, and Jensen's Inequality

Let $C\subseteq \mathbb{R}^d$. The set $C$ is convex when for every $x,y\in C$ and every $\theta\in[0,1]$,

$$\theta x + (1-\theta)y \in C.$$

The vector $\theta x + (1-\theta)y$ is a weighted average of the two points. When $\theta=1$, it is $x$; when $\theta=0$, it is $y$; and for values in between it traces the line segment between them.

Let $f:C\to\mathbb{R}$, where $C$ is convex. The function $f$ is convex when for every $x,y\in C$ and $\theta\in[0,1]$,

$$f\big(\theta x+(1-\theta)y\big) \le \theta f(x)+(1-\theta)f(y).$$

The left side evaluates the function after mixing the inputs. The right side mixes the function values. Their difference,

$$G_\theta(x,y) = \theta f(x)+(1-\theta)f(y) - f\big(\theta x+(1-\theta)y\big),$$

is the Jensen gap for this two-point mixture. Convexity says $G_\theta(x,y)\ge 0$ for every allowed pair and mixture weight.

Jensen's inequality extends the same idea to finitely many points. If $f$ is convex, $x_1,\ldots,x_n\in C$, and weights $\alpha_i\ge0$ satisfy $\sum_{i=1}^n\alpha_i=1$, then

$$f\left(\sum_{i=1}^n \alpha_i x_i\right) \le \sum_{i=1}^n \alpha_i f(x_i).$$

For a random variable $X$ with values in $C$, and with the relevant expectations well-defined, the expectation version is

$$f\big(\mathbb{E}[X]\big) \le \mathbb{E}[f(X)].$$

This is why Jensen appears in probabilistic ML. Moving a convex function outside or inside an average is not neutral; it introduces an inequality direction.

There are two extremely useful equivalent tests.

First, the epigraph of $f$ is

$$\operatorname{epi}(f)=\{(x,t)\in C\times\mathbb{R}: f(x)\le t\}.$$

The function $f$ is convex exactly when $\operatorname{epi}(f)$ is a convex set. Instead of looking at a curve and chords, you can look at the region above the graph.

Second, if $f$ is differentiable on a convex domain, then $f$ is convex exactly when every tangent plane is a global underestimator:

$$f(y)\ge f(x)+\nabla f(x)^\top (y-x) \quad\text{for all }x,y\in C.$$

This is the local-to-global bridge. For a differentiable convex function, any point $x^\star$ with $\nabla f(x^\star)=0$ satisfies

$$f(y)\ge f(x^\star) \quad\text{for all }y\in C.$$

So $x^\star$ is a global minimizer. More generally, convex functions have no non-global local minima. In a nonconvex neural-network objective, the same gradient-zero condition can describe a local minimum, saddle, or maximum; convexity is what makes the local condition globally trustworthy.