Distributions

Let $(\Omega,\mathcal F,P)$ be a probability space. Let $(S,\mathcal S)$ be the measurable space of values, and let

$$X:(\Omega,\mathcal F)\to(S,\mathcal S)$$

be a measurable random variable. The distribution, or law, of $X$ is the probability measure $P_X$ on values:

$$P_X(A)=P(X\in A)=P(\{\omega\in\Omega:X(\omega)\in A\}).$$

Here $A\in\mathcal S$ is a measurable set of values. Equivalently, $P_X(A)=P(X^{-1}(A))$.

This is called the pushforward of $P$ through $X$. It says: start with probability on raw outcomes, apply the measurement $X$, and add up the probability that lands inside each value-set $A$.

For a discrete random variable, the probability mass function is

$$p_X(x)=P(X=x).$$

If the raw sample space is finite, this is just the mass of every outcome that maps to $x$:

$$p_X(x)=\sum_{\omega:X(\omega)=x}P(\omega).$$

This is the central mechanism: a value collects probability from every raw outcome that maps to it. Grouping changes the representation of probability, not the total mass, so $\sum_x p_X(x)=1$.

When $X$ is real-valued, the cumulative distribution function is

$$F_X(t)=P(X\le t).$$

It accumulates the mass up to a threshold. For many real-valued distributions used in machine learning, $P_X$ has a density $f_X$ and interval probabilities are computed by integration:

$$P(a\le X\le b)=\int_a^b f_X(x)\,dx.$$

A density value is not itself a probability; only area under the density over a region is probability. Density values can be greater than $1$, because they depend on the units of $x$. Some distributions are mixed or have no ordinary density, so PMFs and densities are important cases, not the whole definition of distribution.

The demo uses two independent Bernoulli flips with head probability $p$ and random variable $X=$ number of heads. Then $X$ has a binomial distribution:

$$P(X=k)=\binom{2}{k}p^k(1-p)^{2-k},\qquad k\in\{0,1,2\}.$$

Its expectation and variance are

$$\mathbb E[X]=2p,\qquad \operatorname{Var}(X)=2p(1-p).$$

In machine learning, a parametric distribution family writes this mass or density as $p_\theta(x)$. If observed values $x^{(1)},\dots,x^{(n)}$ are treated as independent draws from the distribution, the log likelihood is

$$\ell(\theta)=\sum_{i=1}^n \log p_\theta(x^{(i)}).$$

For discrete data, $p_\theta(x)$ is a probability mass. For continuous data, $p_\theta(x)$ is a density value, so likelihood scores density at the observations; it is not the probability of observing those exact points.