Fisher Information & Information Geometry

Why It Matters for Modern Models

The Fisher matrix gives a family of probability distributions a notion of distance: two parameter settings are close when their distributions are hard to tell apart, however far apart the parameters themselves are (Amari, 2016).
For small updates, a KL-divergence constraint or penalty, as in trust-region policy optimization (TRPO) or the KL penalty used in RLHF, is approximately the quadratic ½ dθᵀF dθ, so it limits how far the distribution moves rather than how far the weights move (Schulman et al., 2015).
It measures how precisely data can pin down the parameters: no unbiased estimator has a covariance smaller than (nF)⁻¹, and under regularity conditions the maximum-likelihood estimator approaches this bound as n grows.

Easy to Miss

The Fisher matrix is the Hessian of KL(p_θ₀ ‖ p_θ) at θ = θ₀, yet it can be computed from first derivatives alone, as the expected outer product of the score, which also makes it positive semidefinite, unlike the Hessian of a training loss.
The Cramér–Rao bound applies to unbiased estimators. A biased estimator, for example one shrunk toward zero, can have a smaller variance than 1/(nF).
The "empirical Fisher" used by some optimizers, built from gradients at the observed labels, is not the Fisher matrix, which averages over labels drawn from the model itself; the two can differ greatly, and the empirical version need not capture curvature (Kunstner et al., 2019).

Core Math

Key Equation

F_{ij}(\theta) = \mathbb{E}\left[\partial_i \log p_\theta \cdot \partial_j \log p_\theta\right]

For a regular family of distributions $p_\theta$ with parameter-independent support and differentiation allowed under the integral, the Fisher information matrix is the covariance of the score $\nabla_\theta \log p_\theta(x)$ , whose mean is zero:

F_{ij}(\theta) = \mathbb{E}_{x \sim p_\theta}\left[\frac{\partial \log p_\theta(x)}{\partial \theta_i} \frac{\partial \log p_\theta(x)}{\partial \theta_j}\right]

Under regularity conditions it also equals the expected negative Hessian of the log-likelihood:

F_{ij}(\theta) = -\mathbb{E}_{x \sim p_\theta}\left[\frac{\partial^2 \log p_\theta(x)}{\partial \theta_i \partial \theta_j}\right]

KL divergence as a local distance. For a small parameter change $d\theta$ ,

\text{KL}(p_\theta \,\|\, p_{\theta + d\theta}) \approx \frac{1}{2}\, d\theta^\top F(\theta)\, d\theta

so $F$ measures how fast the distribution changes, rather than the parameters, as $\theta$ moves.

Cramér–Rao bound. Under these regularity conditions, if $F(\theta)$ is invertible and $\hat\theta$ is an unbiased estimator of $\theta$ computed from $n$ independent samples from $p_\theta$ , then