Cross-Entropy

Let $X$ be a discrete label taking values in $\{1,\dots,K\}$. Let $p_k=P_{\mathrm{target}}(X=k)$ be the target distribution and let $q_k=P_{\theta}(X=k)$ be the model distribution. Assume $p_k\ge 0$, $q_k>0$, and $\sum_k p_k=\sum_k q_k=1$. All logarithms here are natural logarithms, so the units are nats.

For supervised classification, read this as a per-input statement. For an input $x_i$, the target distribution is $p^{(i)}_k=P_{\mathrm{target}}(Y=k\mid x_i)$ and the model distribution is $q^{(i)}_k=P_\theta(Y=k\mid x_i)$. The dataset loss averages $H(p^{(i)},q^{(i)})$ over examples.

The cross-entropy from $p$ to $q$ is

Equivalently,

The direction matters: $p$ supplies the averaging weights, while $q$ supplies the probabilities being scored. If $p_k>0$ and $q_k$ is near zero, the penalty becomes very large. If $p_k>0$ and $q_k=0$, the mathematical loss is infinite. If $p_k=0$, that class does not contribute directly to this cross-entropy term.

For a one-hot target $y$, where $p_y=1$ and all other $p_k=0$,

This is exactly the per-example negative log-likelihood used for multiclass classification and next-token language modeling.

The link to KL divergence is

where

Cross-entropy and forward KL share one support rule: both are infinite when some $p_k>0$ has $q_k=0$. The entropy $H(p)$ does not involve $q$ at all and is finite on a finite set of classes. When the target distribution $p$ is fixed, $H(p)$ is constant with respect to the model. Minimizing cross-entropy over $q_\theta$ is therefore the same optimization problem as minimizing $\mathrm{KL}(p\|q_\theta)$. Maximum likelihood is the empirical version: the data distribution supplies $p$, and the model is trained to reduce the average $-\log q_\theta(x)$ assigned to observed data.

For one-hot targets, the cross-entropy can be pushed arbitrarily close to $0$, but with every $q_k>0$ it never reaches it. For soft targets the floor is $H(p)$, usually above $0$, reached when $q=p$ (and only approached when $p$ has zeros, since softmax keeps every $q_k>0$). In that setting, the KL term is the mismatch and $H(p)$ is the irreducible target uncertainty.

For neural networks, the model usually produces logits $z\in\mathbb R^K$ and probabilities

For the soft-target loss

we can derive the logit gradient directly. Since

and $\sum_i p_i=1$,

Therefore the gradient with respect to each logit is

Classes where $q_k<p_k$ get a negative gradient, so gradient descent increases their logits. Classes where $q_k>p_k$ get a positive gradient, so gradient descent lowers them.

This is a gradient with respect to the logits. With respect to the probabilities themselves it would be $-p_k/q_k$; the tidy $q-p$ appears only after differentiating through the softmax. Its components sum to zero, because softmax can only move probability from one class to another.

The two can point in different directions. With $p=(1/2,1/4,1/4)$ and $q=(1/2,3/8,1/8)$, the probability gradient is $-p/q=(-1,-2/3,-2)$: every probability "should" rise, which no distribution can do. The logit gradient is $q-p=(0,1/8,-1/8)$: raise $C$, lower $B$, leave $A$. Here $H(p)=\tfrac32\log2$, $H(p,q)=2\log2-\tfrac14\log3$ and $\mathrm{KL}(p\|q)=\tfrac14\log\tfrac43\approx0.072$ nats. The reverse cross-entropy $H(q,p)=\tfrac32\log2$ differs from $H(p,q)$: the order of the two distributions matters.

Same accuracy, different loss

Two classifiers can both be right on every example and still differ in loss. Take classes $(A,B,C)$, two examples labelled $A$ and $B$, and two models whose outputs are the rows

Both put the most probability on the right class each time, so both have accuracy $2/2$. Treating the two examples as independent, the likelihood of the labels is the product of the probabilities given to them: $\tfrac12\cdot\tfrac34=\tfrac38$ for $M$ and $\tfrac23\cdot\tfrac12=\tfrac13$ for $N$. The mean one-hot cross-entropy is $\tfrac12\log\tfrac83\approx0.490$ nats for $M$ and $\tfrac12\log3\approx0.549$ for $N$. Taking logs keeps the order and the minus sign reverses it, so the higher likelihood and the lower cross-entropy pick the same model, $M$, whose confidence went to what actually happened. Two examples, of course, say nothing about either model on new data.

import numpy as np

def softmax(logits):
    shifted = logits - logits.max()
    exp = np.exp(shifted)
    return exp / exp.sum()

def log_softmax(logits):
    shifted = logits - logits.max()
    return shifted - np.log(np.exp(shifted).sum())

def entropy(p):
    p = np.asarray(p, dtype=float)
    mask = p > 0
    return float(-np.sum(p[mask] * np.log(p[mask])))

def cross_entropy_from_logits(p, logits):
    p = np.asarray(p, dtype=float)
    log_q = log_softmax(logits)
    return float(-np.sum(p * log_q))

def finite_difference_grad(loss_fn, logits, eps=1e-6):
    logits = np.asarray(logits, dtype=float)
    out = np.zeros_like(logits)
    for k in range(logits.size):
        plus = logits.copy()
        minus = logits.copy()
        plus[k] += eps
        minus[k] -= eps
        out[k] = (loss_fn(plus) - loss_fn(minus)) / (2 * eps)
    return out

# Shapes: p, logits, q, and grad_logits are all (K,).
p = np.array([0.70, 0.20, 0.08, 0.02])
logits = np.array([1.2, 0.4, -0.3, -1.1])

log_q = log_softmax(logits)
q = softmax(logits)
ce = cross_entropy_from_logits(p, logits)
h = entropy(p)
kl = ce - h
grad_logits = q - p
numeric_grad = finite_difference_grad(lambda z: cross_entropy_from_logits(p, z), logits)

print("q:", np.round(q, 3))
print("H(p,q):", round(ce, 4))
print("H(p):", round(h, 4))
print("KL(p||q):", round(kl, 4))
print("gradient wrt logits:", np.round(grad_logits, 3))
print("finite-difference grad:", np.round(numeric_grad, 3))
print("gradient check:", np.allclose(grad_logits, numeric_grad, atol=1e-6))

# For a one-hot target y=0, the same formula becomes NLL.
y = 0
one_hot = np.eye(4)[y]
print("one-hot CE:", round(cross_entropy_from_logits(one_hot, logits), 4))
print("-log q_y:", round(float(-log_q[y]), 4))

p is a soft target and q the softmax of the logits; these are different numbers from the figure, so that no target weight is zero. The printout shows H(p,q) splitting into H(p) + KL(p||q), and a finite-difference check confirming that the logit gradient is q - p.

The loss is exact; what it buys is less settled. Next-token cross-entropy falls smoothly, as a power law, as language models grow in size, data and compute (Kaplan et al., 2020). Later fits add a floor the loss approaches but never crosses (Hoffmann et al., 2022), read as the entropy of text itself: the $H(p)$ of language, which nobody can measure directly. Shannon (1951) could only bound it for English. Whether a smooth fall in loss means a smooth rise in ability is still argued (Schaeffer et al., 2023). And although the loss is minimised by honest probabilities, large networks trained on it are often overconfident (Guo et al., 2017), so how best to keep a model's confidence honest remains an open question.