Domain Neighborhood

Information Theory

How we measure information and mismatch between distributions: entropy, cross-entropy, KL divergence, mutual information, and why they appear everywhere in ML.

1 concepts1 published1 demos

Start with KL Divergence (Relative Entropy)Search Atlas

Selected domain objectKL Divergence (Relative Entropy)

Start here. Predict once, then carry the invariant forward.

p(x)q(x)log p/qKL

Open first notebook

QuestionWhich invariant should survive into Information Theory?

PredictionBefore the first demo, predict which variable moves first.

Evidence1 demo witnesses in this domain

InvariantName the mechanism before continuing the route.

Learner lensWhat makes this domain feel navigable?

Stabilize the first mechanism, make one prediction, then move one node forward.

Take this lens

Recommended Route

Start here, then follow the prerequisites forward.

This sequence is ordered for learning rather than inventory. Published notebooks with an unavailable prerequisite—or a same-domain route step that depends on one—are labeled in the full inventory instead of being presented as ready steps.

01
KL Divergence (Relative Entropy)
KL divergence is a directional expected log-probability mismatch between distributions; it explains cross-entropy training, variational inference, and KL-regularized alignment.
14 mincodedemoafter Distributions, Cross-Entropy
Check Distributions first if the symbols feel slippery.

All Published Notebooks

Browse the territory.

KL Divergence (Relative Entropy)

KL divergence is a directional expected log-probability mismatch between distributions; it explains cross-entropy training, variational inference, and KL-regularized alignment.

Level 314 mindemo