Maximum Likelihood

Let $X_1,\dots,X_n$ be observed values treated as independent draws from a fixed parametric model family $p_\theta(x)$. The model family and parameter space are chosen before fitting; maximum likelihood only moves $\theta$ inside that family. All logarithms below are natural logarithms, so losses are measured in nats.

The likelihood is a function of the parameter:

The data values are fixed inside this expression. The variable being optimized is $\theta$. Because products of many probabilities become tiny, we usually maximize log likelihood:

Equivalently, training minimizes average negative log-likelihood:

For a Bernoulli model with $x_i\in\{0,1\}$, parameter space $\theta\in[0,1]$, and $P_\theta(X=1)=\theta$, suppose $s$ observations are $1$ and $f=n-s$ are $0$. For an ordered sequence,

and

If the data record only the count $s$ rather than the ordered sequence, the likelihood also has a binomial coefficient $\binom n s$. That factor does not depend on $\theta$, so it does not change the MLE.

On the open interval $0<\theta<1$, the derivative is

When $0<s<n$, setting it to zero gives

When $s=0$ or $s=n$, there is no interior critical point. On the closed interval $[0,1]$, the MLE is the boundary value $\hat\theta_{\mathrm{MLE}}=0$ or $\hat\theta_{\mathrm{MLE}}=1$. On the open interval $(0,1)$, the maximum is not attained; the likelihood only approaches its supremum at the boundary. The demo displays $\theta\in[0.01,0.99]$ to avoid infinities from $\log 0$.

This is not a coincidence. The MLE for this Bernoulli family is the empirical frequency because the best one-parameter Bernoulli distribution matches the observed mass on $1$ and $0$.

Now write the empirical distribution as

In this finite discrete setting, the average NLL is the cross-entropy from the empirical distribution to the model distribution:

And cross-entropy decomposes as

Here the empirical entropy is

and the forward KL mismatch is

The sums are over $x\in\{0,1\}$. Terms with $\hat p(x)=0$ contribute $0$. If $\hat p(x)>0$ but $p_\theta(x)=0$, the NLL and KL are infinite.

Since $H(\hat p)$ does not depend on $\theta$, maximum likelihood is equivalent here to minimizing the KL mismatch. If the model family cannot represent the empirical distribution exactly, MLE chooses the member of the family with the smallest mismatch inside that family.

The derivative of the average Bernoulli NLL with respect to $\theta$ is

If $\theta$ is produced by a logit $a$ with $\theta=\sigma(a)$, then the chain rule gives

The demo reports this logit gradient. It is not the slope of the plotted curve with respect to $\theta$.

For neural classifiers, $p_\theta(y\mid x)$ is conditional on the input. The dataset objective is

For language models, $y_i$ is the next token at a position. The mechanism is the same: assign high probability to observed data, take logs, average, then use gradients to move parameters.

For continuous models, $p_\theta(x)$ is a density rather than a probability mass. The likelihood scores density at the observed points; the probability of any exact point can be zero even while the likelihood density is high. Likelihood comparisons are meaningful within the same model and measurement units. The finite-sample objective is an empirical average of log density; a KL interpretation is clean when comparing expected NLL under a data-generating density to model densities with respect to the same base measure.

A three-toss example

Take the ordered sequence $x=[1,1,0]$, so $L(\theta)=\theta^2(1-\theta)$. Two candidates give $L(1/2)=1/8$ and $L(3/4)=9/64$; since $9/64>8/64$, $\theta=3/4$ makes these data more probable, and because $\log$ keeps the order of positive numbers, $\ell$ and the negative log-likelihood rank the two the same way. The derivative $\ell'(\theta)=2/\theta-1/(1-\theta)=(2-3\theta)/\big(\theta(1-\theta)\big)$ vanishes at $\hat\theta=2/3$, and $\ell''(\theta)=-2/\theta^2-1/(1-\theta)^2<0$ on $(0,1)$, so this is the only maximum, with $L(2/3)=4/27$. Recording only the count $s=2$ multiplies every value by $\binom32=3$ and leaves the maximum where it is.

The likelihood is not a probability distribution over $\theta$:

So $L(2/3)=4/27$ is the probability of the observed sequence when $\theta=2/3$, not the probability that $\theta=2/3$. Change only the data, to $x=[1,0,0,0]$: now $L(\theta)=\theta(1-\theta)^3$, the maximum moves to $\hat\theta=1/4$ and $L(1/4)=27/256$. The fitting principle is the same; the data changed.

For long sequences, compute $\ell(\theta)=s\log\theta+f\log(1-\theta)$ directly rather than as the log of the product. A product of many probabilities is too small for floating point: with $1{,}100$ tosses at $\theta=1/2$, $L=2^{-1100}$ rounds to $0$ in double precision, while $\ell=-1100\log2\approx-762.46$ is an ordinary number.