Random Variables

Let $(\Omega,\mathcal F,P)$ be a probability space. The sample space $\Omega$ contains raw outcomes $\omega$, the event collection $\mathcal F$ says which subsets can be assigned probabilities, and $P$ assigns probability to those events.

A real-valued random variable is a measurable function

Measurable means that whenever $B$ is a Borel set of possible values, the set of raw outcomes that land in $B$ is an event:

The distribution, or law, of $X$ is the pushforward of $P$ through $X$. For any Borel set $B\in\mathcal B(\mathbb R)$,

In the finite discrete case, write $P(\omega)$ as shorthand for the atomic mass $P(\{\omega\})$. Then all outcomes mapping to the same value get grouped together:

The support of $X$ in this finite setting is

The group of outcomes mapping to one value is the fiber over $x$:

The probability mass of that fiber becomes the PMF value $p_X(x)$.

This is the bridge to the next concept, distributions: the random variable is the map, and the distribution is the probability mass after the map.

The expectation can be computed over raw outcomes or over the induced distribution:

Variance measures squared spread around that expectation:

When $X$ has a density $f_X$, probabilities and expectations are computed by integration:

A density value is not itself a probability, and not every distribution has an ordinary density. The measure $P_X$ is the general object; PMFs and densities are common representations of it.

In machine learning, a common random variable is the loss on a sampled example. If $Z$ is a random training example and $\theta$ are model parameters, then

is a random variable because the sampled example $Z$ is uncertain. Training tries to reduce $\mathbb{E}[L]$, while a mini-batch computes a noisy estimate:

For independent sampled examples, $\mathbb E[\hat L_m]=\mathbb E[L]$ and $\operatorname{Var}(\hat L_m)=\operatorname{Var}(L)/m$. That is one reason batch size, learning rate, and optimization stability are connected.

This is also the bridge to likelihood and Bayesian inference. Observed data are values of random variables under a model distribution; likelihood scores those observed values, cross-entropy averages their surprise, and Bayesian inference uses likelihoods to update distributions over unknowns.