KL Divergence (Relative Entropy)

Let $p$ and $q$ be probability distributions on the same finite or countable outcome space $\mathcal X$. The KL divergence from $p$ to $q$ is

Equivalently,

Logarithms are natural, so the units are nats. Terms with $p(x)=0$ contribute $0$. If $p(x)>0$ and $q(x)=0$ for any outcome, then $\mathrm{KL}(p\|q)=\infty$: the model has ruled out something the data does.

KL is nonnegative:

with equality exactly when $p=q$ as distributions, up to events with probability zero.

To see why, let $R=q(X)/p(X)$ for outcomes where $p(X)>0$. Since $\log$ is concave,

But $\mathbb E_p[R]=\sum_{x:p(x)>0}q(x)\le 1$, so $\mathbb E_p[\log R]\le 0$. Multiplying by $-1$ gives $\mathrm{KL}(p\|q)\ge 0$. Single terms can be negative; the total cannot.

KL is generally asymmetric:

The asymmetry comes from the expectation. In $\mathrm{KL}(p\|q)$, samples are drawn from $p$. In $\mathrm{KL}(q\|p)$, samples are drawn from $q$. Changing the averaging distribution changes which mistakes are visited often. Which one goes first is a choice: in supervised training it is the data; in the penalty $\mathrm{KL}(\pi_\theta\|\pi_{\mathrm{ref}})$ used when fine-tuning a policy, it is the model being trained.

The cross-entropy identity is

Here

and

The identity is just the pointwise equality

averaged under $p$. With $p$ fixed, minimizing cross-entropy over $q_\theta$ is the same as minimizing $\mathrm{KL}(p\|q_\theta)$, because $H(p)$ does not depend on the model. This is the bridge to maximum likelihood: the data define $p$, the model supplies $q_\theta$.

For continuous variables with densities $p(x)$ and $q(x)$ relative to the same base measure,

It is still an average of a log ratio, taken under the first distribution.

Support first

On a finite set of outcomes, $\mathrm{KL}(p\|q)$ is finite exactly when $q(x)>0$ wherever $p(x)>0$, so the first thing to check is support. Take $p=(1/2,1/4,1/4)$ on three outcomes. If $q_0=(3/4,1/4,0)$, then $\mathrm{KL}(p\|q_0)=\infty$, because $p$ produces the third outcome and $q_0$ rules it out, while $\mathrm{KL}(q_0\|p)=\tfrac34\log\tfrac32\approx0.304$ is finite, because $q_0$ never visits that outcome. Replace the zero by a near miss, $q_\varepsilon=(3/4,3/16,1/16)$, and both directions become finite:

The first term of the forward sum is negative, $\tfrac12\log\tfrac23\approx-0.203$, where $q_\varepsilon$ overrates the first outcome; the total is not. The forward direction happens to be the larger one here; that is a property of these numbers, not a rule about the two directions.

Beyond finite sets, the condition becomes absolute continuity: $p$ must give probability zero to every event that $q$ does. That is necessary for a finite divergence but not sufficient. On the positive integers, take $p(k)=6/(\pi^2k^2)$ and $q(k)=2^{-k}$. Every $q(k)$ is positive, yet $\log\big(p(k)/q(k)\big)$ grows like $k\log2$, and $\sum_k p(k)\,k$ diverges, so $\mathrm{KL}(p\|q)=\infty$.

import math
import numpy as np

def check_distribution(v):
    v = np.asarray(v, dtype=float)
    assert np.all(v >= 0)
    assert abs(v.sum() - 1.0) < 1e-12
    return v

def entropy(p):
    p = check_distribution(p)
    return float(sum(-px * math.log(px) for px in p if px > 0))

def cross_entropy(p, q):
    p = check_distribution(p)
    q = check_distribution(q)
    assert p.shape == q.shape
    if np.any((p > 0) & (q == 0)):
        return math.inf
    return float(sum(-px * math.log(qx) for px, qx in zip(p, q) if px > 0))

def kl(p, q):
    p = check_distribution(p)
    q = check_distribution(q)
    assert p.shape == q.shape
    if np.any((p > 0) & (q == 0)):
        return math.inf
    return float(sum(px * (math.log(px) - math.log(qx)) for px, qx in zip(p, q) if px > 0))

# Shapes: p and q are length-K categorical distributions.
# Supervised reading: p=data/target, q=model.
# Policy-regularization reading for KL(q||p): q=new policy, p=reference.
p = [0.55, 0.25, 0.15, 0.05]
q = [0.80, 0.15, 0.04, 0.01]

h_p = entropy(p)
h_pq = cross_entropy(p, q)
kl_pq = kl(p, q)
kl_qp = kl(q, p)

assert abs(h_pq - (h_p + kl_pq)) < 1e-12

print("KL(p || q):", round(kl_pq, 4))
print("KL(q || p):", round(kl_qp, 4))
print("H(p,q):", round(h_pq, 4))
print("H(p)+KL(p||q):", round(h_p + kl_pq, 4))

It prints 0.2004 and 0.1542 for the two directions, and 1.3101 twice: cross-entropy is entropy plus KL. When $q$ gives zero probability to an outcome $p$ can produce, kl returns infinity rather than failing.

Which direction to minimise is a modelling choice with real consequences, and there is no single right answer. Fitting $Q$ by minimising $\mathrm{KL}(P\|Q)$, as maximum likelihood does, spreads $Q$ to cover everything $P$ does. Minimising $\mathrm{KL}(Q\|P)$, as standard variational inference does, lets $Q$ lock onto one mode and understate uncertainty (Bishop, 2006, §10.1; Blei, Kucukelbir & McAuliffe, 2017). Variational inference uses the reverse direction largely because it can be computed without the true posterior, not because it is the better fit. Language-model fine-tuning with human feedback adds a penalty $\mathrm{KL}(\pi\|\pi_{\text{ref}})$ that keeps the new policy near the model it started from (Ouyang et al., 2022). How strong that leash should be, what it protects, and what it lets through are still being worked out.