Maximum Likelihood

Let $X_1,\dots,X_n$ be observed values treated as independent draws from a fixed parametric model family $p_\theta(x)$. The model family and parameter space are chosen before fitting; maximum likelihood only moves $\theta$ inside that family. All logarithms below are natural logarithms, so losses are measured in nats.

The likelihood is a function of the parameter:

$$L(\theta)=\prod_{i=1}^n p_\theta(x_i).$$

The data values are fixed inside this expression. The variable being optimized is $\theta$. Because products of many probabilities become tiny, we usually maximize log likelihood:

$$\ell(\theta)=\log L(\theta)=\sum_{i=1}^n \log p_\theta(x_i).$$

Equivalently, training minimizes average negative log-likelihood:

$$\mathcal L_{\mathrm{NLL}}(\theta)=-\frac{1}{n}\sum_{i=1}^n \log p_\theta(x_i).$$

For a Bernoulli model with $x_i\in\{0,1\}$, parameter space $\theta\in[0,1]$, and $P_\theta(X=1)=\theta$, suppose $s$ observations are $1$ and $f=n-s$ are $0$. For an ordered sequence,

$$L(\theta)=\theta^s(1-\theta)^f,$$

and

$$\ell(\theta)=s\log\theta+f\log(1-\theta).$$

If the data record only the count $s$ rather than the ordered sequence, the likelihood also has a binomial coefficient $\binom n s$. That factor does not depend on $\theta$, so it does not change the MLE.

On the open interval $0<\theta<1$, the derivative is

$$\frac{d\ell}{d\theta}=\frac{s}{\theta}-\frac{f}{1-\theta}.$$

When $0<s<n$, setting it to zero gives

$$\hat\theta_{\mathrm{MLE}}=\frac{s}{n}.$$

When $s=0$ or $s=n$, there is no interior critical point. On the closed interval $[0,1]$, the MLE is the boundary value $\hat\theta_{\mathrm{MLE}}=0$ or $\hat\theta_{\mathrm{MLE}}=1$. On the open interval $(0,1)$, the maximum is not attained; the likelihood only approaches its supremum at the boundary. The demo displays $\theta\in[0.01,0.99]$ to avoid infinities from $\log 0$.

This is not a coincidence. The MLE for this Bernoulli family is the empirical frequency because the best one-parameter Bernoulli distribution matches the observed mass on $1$ and $0$.

Now write the empirical distribution as

$$\hat p(1)=\frac{s}{n},\qquad \hat p(0)=\frac{f}{n}.$$

In this finite discrete setting, the average NLL is the cross-entropy from the empirical distribution to the model distribution:

$$\mathcal L_{\mathrm{NLL}}(\theta)=H(\hat p,p_\theta).$$ $$H(\hat p,p_\theta)=-\sum_{x\in\{0,1\}}\hat p(x)\log p_\theta(x).$$

And cross-entropy decomposes as

$$H(\hat p,p_\theta)=H(\hat p)+\mathrm{KL}(\hat p\|p_\theta).$$

Here the empirical entropy is

$$H(\hat p)=-\sum_x \hat p(x)\log\hat p(x),$$

and the forward KL mismatch is

$$\mathrm{KL}(\hat p\|p_\theta)=\sum_x \hat p(x)\log\frac{\hat p(x)}{p_\theta(x)}.$$

The sums are over $x\in\{0,1\}$. Terms with $\hat p(x)=0$ contribute $0$. If $\hat p(x)>0$ but $p_\theta(x)=0$, the NLL and KL are infinite.

Since $H(\hat p)$ does not depend on $\theta$, maximum likelihood is equivalent here to minimizing the KL mismatch. If the model family cannot represent the empirical distribution exactly, MLE chooses the member of the family with the smallest mismatch inside that family.

The derivative of the average Bernoulli NLL with respect to $\theta$ is

$$\frac{d\mathcal L}{d\theta}=-\frac{\hat p}{\theta}+\frac{1-\hat p}{1-\theta}.$$

If $\theta$ is produced by a logit $a$ with $\theta=\sigma(a)$, then the chain rule gives

$$\frac{d\mathcal L}{da}=\theta-\hat p.$$

The demo reports this logit gradient. It is not the slope of the plotted curve with respect to $\theta$.

For neural classifiers, $p_\theta(y\mid x)$ is conditional on the input. The dataset objective is

$$-\frac{1}{n}\sum_{i=1}^n \log p_\theta(y_i\mid x_i).$$

For language models, $y_i$ is the next token at a position. The mechanism is the same: assign high probability to observed data, take logs, average, then use gradients to move parameters.

For continuous models, $p_\theta(x)$ is a density rather than a probability mass. The likelihood scores density at the observed points; the probability of any exact point can be zero even while the likelihood density is high. Likelihood comparisons are meaningful within the same model and measurement units. The finite-sample objective is an empirical average of log density; a KL interpretation is clean when comparing expected NLL under a data-generating density to model densities with respect to the same base measure.