Cross-Entropy

Let $X$ be a discrete label taking values in $\{1,\dots,K\}$. Let $p_k=P_{\mathrm{target}}(X=k)$ be the target distribution and let $q_k=P_{\theta}(X=k)$ be the model distribution. Assume $p_k\ge 0$, $q_k>0$, and $\sum_k p_k=\sum_k q_k=1$. All logarithms here are natural logarithms, so the units are nats.

For supervised classification, read this as a per-input statement. For an input $x_i$, the target distribution is $p^{(i)}_k=P_{\mathrm{target}}(Y=k\mid x_i)$ and the model distribution is $q^{(i)}_k=P_\theta(Y=k\mid x_i)$. The dataset loss averages $H(p^{(i)},q^{(i)})$ over examples. The demo below shows one such example.

The cross-entropy from $p$ to $q$ is

$$H(p,q)=-\sum_{k=1}^K p_k \log q_k.$$

Equivalently,

$$H(p,q)=\mathbb E_{X\sim p}[-\log q_X].$$

The direction matters: $p$ supplies the averaging weights, while $q$ supplies the probabilities being scored. If $p_k>0$ and $q_k$ is near zero, the penalty becomes very large. If $p_k>0$ and $q_k=0$, the mathematical loss is infinite. If $p_k=0$, that class does not contribute directly to this cross-entropy term.

For a one-hot target $y$, where $p_y=1$ and all other $p_k=0$,

$$H(p,q)=-\log q_y.$$

This is exactly the per-example negative log-likelihood used for multiclass classification and next-token language modeling.

The link to KL divergence is

$$H(p,q)=H(p)+\mathrm{KL}(p\|q),$$

where

$$H(p)=-\sum_k p_k\log p_k,\qquad \mathrm{KL}(p\|q)=\sum_k p_k\log\frac{p_k}{q_k}.$$

When the target distribution $p$ is fixed, $H(p)$ is constant with respect to the model. Minimizing cross-entropy over $q_\theta$ is therefore the same optimization problem as minimizing $\mathrm{KL}(p\|q_\theta)$. Maximum likelihood is the empirical version: the data distribution supplies $p$, and the model is trained to reduce the average $-\log q_\theta(x)$ assigned to observed data.

For one-hot targets, the minimum possible cross-entropy is $0$. For soft targets, the minimum possible cross-entropy is usually not $0$; it is $H(p)$, achieved when $q=p$. In that setting, the KL term is the mismatch and $H(p)$ is the irreducible target uncertainty.

For neural networks, the model usually produces logits $z\in\mathbb R^K$ and probabilities

$$q_k=\mathrm{softmax}(z)_k=\frac{e^{z_k}}{\sum_j e^{z_j}}.$$

For the soft-target loss

$$\ell(z,p)=-\sum_k p_k\log(\mathrm{softmax}(z)_k),$$

we can derive the logit gradient directly. Since

$$\log(\mathrm{softmax}(z)_i)=z_i-\log\sum_j e^{z_j},$$

and $\sum_i p_i=1$,

$$\ell(z,p)=-\sum_i p_i z_i+\log\sum_j e^{z_j}.$$

Therefore the gradient with respect to each logit is

$$\frac{\partial \ell}{\partial z_k}=q_k-p_k.$$

This compact gradient is one reason cross-entropy is so useful. Classes where $q_k<p_k$ get a negative gradient, so gradient descent increases their logits. Classes where $q_k>p_k$ get a positive gradient, so gradient descent lowers them.

This is a gradient with respect to logits, not with respect to probabilities. If $q$ were treated as an unconstrained probability vector, then $\partial H(p,q)/\partial q_k=-p_k/q_k$. The formula $q-p$ appears after the loss is differentiated through the softmax. Its components sum to zero, reflecting that softmax logits redistribute probability mass across classes.

This page is about categorical cross-entropy for mutually exclusive classes. Multi-label problems usually use sigmoid outputs and a sum of binary cross-entropies instead.