Continuous Function

Domain

Information Theory

How information and the mismatch between distributions are measured: entropy, cross-entropy and KL divergence, and why they appear throughout machine learning.

1 notebook1 demo
Start with KL Divergence (Relative Entropy)Search concepts
Preview the recommended route
First notebookKL Divergence (Relative Entropy)

Start here. Make one prediction, then note what carries over to the next notebook.

p(x)q(x)log p/qKL
Open first notebook
QuestionWhat does KL Divergence establish for the rest of Information Theory?
PredictionBefore opening the first demo, predict which quantity changes first.
Demos1 interactive demo in this domain
Before moving onSay in one sentence how the idea works before going to the next notebook.
LearnerWhich notebook should I read first?

Read the first notebook, make one prediction in its demo, then go on to the next.

Open the first notebook

Recommended route

Read these in order: each notebook comes after the notebooks in this domain that it builds on, and the label beside it gives its level. A prerequisite with no page here yet is marked as outside reading. A notebook is left out of the route only when a prerequisite has neither a page here nor a named outside reading; the full list below marks it.

  1. 01
    KL Divergence (Relative Entropy)

    KL divergence is the average log-probability ratio between two distributions, taken under the first. Training with cross-entropy minimizes it, variational inference minimizes it in the other direction, and KL-regularized fine-tuning uses it as a penalty.

    Intermediate14 mincodedemoafter Distributions, Cross-Entropy

All notebooks

Every published notebook in this domain.

KL Divergence (Relative Entropy)

KL divergence is the average log-probability ratio between two distributions, taken under the first. Training with cross-entropy minimizes it, variational inference minimizes it in the other direction, and KL-regularized fine-tuning uses it as a penalty.

Intermediate14 mindemo