Foundation Lab

Maximum Likelihood, Cross-Entropy & KL Divergence

Pretraining for GPT-4, Claude, Gemini, Llama: next-token cross-entropy over web/text/code

Concept 1 of 100Core TrainingPhase 1

Open New Notebook All Foundations

#1ML/CE/KLCore Training

key equation\min_\theta H(\hat p, p_\theta) = -\mathbb E_{x\sim \hat p} \log p_\theta(x)

Selected Foundation Object

Keep the equation fixed; move through the evidence.

Concept 1 of 100ML/CE/KLCore Training / Phase 1: Core probabilistic training + transformers

Current question

Why KL direction matters (KL(p_data || p_θ) vs reverse) and how it biases models toward covering modes vs being conservative

\min_\theta H(\hat p, p_\theta) = -\mathbb E_{x\sim \hat p} \log p_\theta(x)

PredictionCommit before the demo.

Ask what should change when the equation is manipulated, then let the visualization test that expectation.

EvidenceCompare local witness and source.

Use the runnable panel, the key equation, and canonical papers as separate forms of evidence for the same object.

InvariantName what survives notation changes.

The useful learning product is the reusable mechanism you can carry into another model, paper, or engineering tradeoff.

Next moveContinue in the newer notebook.

This atlas page keeps the working demo; the domain notebook carries the fuller Intuition -> Math -> Code -> Demo sequence.

Open demo Inspect equation Open newer notebook Next foundation

Why It Matters for Modern Models

Pretraining for GPT-4, Claude, Gemini, Llama: next-token cross-entropy over web/text/code
Stable Diffusion & Sora optimize likelihood-style surrogates (noise-prediction MSE = reparameterized ELBO)
Reward models in RLHF trained via cross-entropy on human preference data

What Tutorials Skip

What is still poorly explained in textbooks and papers:

Why KL direction matters (KL(p_data || p_θ) vs reverse) and how it biases models toward covering modes vs being conservative
How cross-entropy shapes behavior under distribution shift (hallucinations: model picks "likely token" under learned p_θ even when input is off-manifold)

Interactive Visualization

Core Math (Optional Deep Dive)

If you want intuition first, start with the key equation and the visualization. Come back here for the full walkthrough.

Key Equation

\min_\theta H(\hat p, p_\theta) = -\mathbb E_{x\sim \hat p} \log p_\theta(x)

Almost every frontier model is trained by (approximate) maximum likelihood:

\max_\theta \sum_{i=1}^n \log p_\theta(x^{(i)})

Equivalently, minimize empirical cross-entropy between data distribution $\hat p$ and model $p_\theta$ :

\min_\theta H(\hat p, p_\theta) = \min_\theta \left[ -\mathbb E_{x\sim \hat p} \log p_\theta(x) \right]

This is the same as minimizing KL divergence:

\mathrm{KL}(\hat p \,\|\, p_\theta) = \mathbb E_{\hat p} \log \frac{\hat p(x)}{p_\theta(x)}

For autoregressive LMs, factorization comes from the chain rule:

p_\theta(x_1,\dots,x_T) = \prod_{t=1}^T p_\theta(x_t \mid x_{<t})

Canonical Papers

A Neural Probabilistic Language Model

Bengio et al.2003JMLR

Read paper →

Connections

Next Moves

Choose the next question to carry this object forward.

Semantic Connections

Reuse / Same Technique

⚖Cross-entropy→RLHF ℤKL divergence→VAEs 🔤Viterbi segmentation→Tokens 🎯Likelihood ratios→DPO ⚔MLE-trained classifier→GANs

Dual / Mathematical Dual

∀Likelihood ↔ MDL→Theory ∂Denoising ↔ ELBO→Diffusion

Fix / Invented to Fix

∇Efficient MLE→Adam 🎲Fix text degeneration→Decoding

Break / Breaks When

⚔Non-overlapping supports→GANs 🔤OOV tokens→Tokens 🎲Repetition loops→Decoding