KL Divergence (Relative Entropy)

Let $p$ and $q$ be probability distributions on the same finite or countable outcome space $\mathcal X$. The KL divergence from $p$ to $q$ is

$$\mathrm{KL}(p\|q)=\sum_{x\in\mathcal X}p(x)\log\frac{p(x)}{q(x)}.$$

Equivalently,

$$\mathrm{KL}(p\|q)=\mathbb E_{X\sim p}[\log p(X)-\log q(X)].$$

All logarithms here are natural logs, so the units are nats. Terms with $p(x)=0$ contribute $0$. If $p(x)>0$ and $q(x)=0$, then $\mathrm{KL}(p\|q)=\infty$. That support condition is not a technical footnote; it is the source of the "missing data mode" failure.

KL is nonnegative:

$$\mathrm{KL}(p\|q)\ge 0,$$

with equality exactly when $p=q$ as distributions, up to events with probability zero.

One quick way to see the nonnegativity is to let $R=q(X)/p(X)$ for outcomes where $p(X)>0$. Since $\log$ is concave,

$$\mathbb E_p[\log R]\le \log \mathbb E_p[R].$$

But $\mathbb E_p[R]=\sum_{x:p(x)>0}q(x)\le 1$, so $\mathbb E_p[\log R]\le 0$. Multiplying by $-1$ gives $\mathrm{KL}(p\|q)\ge 0$. This is why individual signed terms can be negative while the total divergence is never negative.

KL is generally asymmetric:

$$\mathrm{KL}(p\|q)\ne\mathrm{KL}(q\|p).$$

The asymmetry comes from the expectation. In $\mathrm{KL}(p\|q)$, samples are drawn from $p$. In $\mathrm{KL}(q\|p)$, samples are drawn from $q$. Changing the averaging distribution changes which mistakes are visited often.

The cross-entropy identity is

$$H(p,q)=H(p)+\mathrm{KL}(p\|q).$$

Here

$$H(p,q)=-\sum_x p(x)\log q(x),$$

and

$$H(p)=-\sum_x p(x)\log p(x).$$

The identity is just the pointwise equality

$$\begin{aligned} -\log q(x) &= -\log p(x) \\ &\quad + \log\frac{p(x)}{q(x)} \end{aligned}$$

averaged under $p$. If $p$ is fixed, minimizing cross-entropy over $q_\theta$ is the same as minimizing $\mathrm{KL}(p\|q_\theta)$, because $H(p)$ does not depend on the model. This is the clean finite-discrete version of the maximum-likelihood bridge: empirical data define $p$, the model supplies $q_\theta$, and training punishes low probability on observed outcomes.

For continuous variables with densities $p(x)$ and $q(x)$ relative to the same base measure,

$$\mathrm{KL}(p\|q)=\int p(x)\log\frac{p(x)}{q(x)}\,dx.$$

Density values are not probabilities by themselves, and the common-reference-measure assumption matters. The practical lesson remains the same: KL compares log density ratios under one chosen averaging distribution.