Reproducing Kernel Hilbert Spaces

Why It Matters for Modern Models

A reproducing kernel Hilbert space is a space of functions in which evaluating a function at a point x is an inner product with the kernel centered at x. Support vector machines, kernel ridge regression and Gaussian-process regression all fit predictors that are finite combinations of kernels centered on the training points (for Gaussian-process regression, the posterior mean), plus any explicit intercept or prior mean. The coefficients are computed from the n × n matrix of kernel values on the training data (Schölkopf & Smola, 2002).
It explains why a similarity function used as a kernel must be symmetric and positive semidefinite: those are exactly the conditions under which it is an inner product in some feature space.
In the infinite-width limit, a network trained by gradient descent behaves like kernel regression with the neural tangent kernel, so RKHS tools describe that limit (Jacot, Gabriel & Hongler, 2018).

Questions to Think About

Why is positive semidefiniteness exactly the condition for a function k(x, x′) to be an inner product of some feature map?
Attention can be read as a kernel smoother over tokens, with weights proportional to exp(qᵀk/√d); why is that kernel generally not symmetric, and so not a reproducing kernel in the strict sense (Tsai et al., 2019)?

Core Math

Key Equation

f(x) = \langle f, k(x, \cdot) \rangle_{\mathcal{H}}

A reproducing kernel Hilbert space $\mathcal{H}$ is a space of functions in which evaluating a function at a point is an inner product with a fixed function $k(x, \cdot)$ , the kernel centred at $x$ :

f(x) = \langle f, k(x, \cdot) \rangle_{\mathcal{H}} \quad \text{for all } f \in \mathcal{H}

This is the reproducing property. Every symmetric, positive semidefinite kernel $k$ defines exactly one such space (the Moore–Aronszajn theorem; Aronszajn, 1950).

Kernel trick. The kernel is an inner product of feature vectors, $k(x, x') = \langle \varphi(x), \varphi(x') \rangle$ , so an algorithm that uses only inner products can work in a very high- or infinite-dimensional feature space while computing only kernel values.

Mercer decomposition. A continuous kernel on a compact domain has an expansion in eigenfunctions $e_m$ with non-negative eigenvalues $\lambda_m$ :

k(x, x') = \sum_{m=1}^\infty \lambda_m e_m(x) e_m(x')

Representer theorem. If the training objective is a loss on the training points plus an increasing function of $\|f\|_{\mathcal{H}}$ , a minimizer is a combination of kernels centred at the $n$ training points (Schölkopf & Smola, 2002):