Probability Basics

For this page, start with a finite probability model. A probability space is $(\Omega,\mathcal F,P)$:

• $\Omega$: sample space (all outcomes)
• $\mathcal{F}$: events (in a finite model, usually all subsets of $\Omega$; in general, a sigma-algebra of measurable subsets)
• $P$: probability measure

For events $E\in\mathcal F$, the basic axioms are:

1. $P(E) \ge 0$
2. $P(\Omega)=1$
3. If $E_1,E_2,\dots$ are pairwise disjoint, then

For finite examples, this reduces to adding the probabilities of disjoint pieces.

The complement rule follows:

Conditional probability means updating probabilities after observing an event $O$:

The denominator matters. Conditioning does not merely keep $E\cap O$; it rescales all probability inside $O$ so the new total mass is $1$. If $P(O)=0$, there is no mass inside the observed event to renormalize, so $P(E\mid O)$ is undefined.

The product rule is the same equation rearranged:

when the conditioning events have positive probability.

Independence means learning one event does not change the probability of the other:

Equivalently, when $P(O)>0$, independence means $P(E\mid O)=P(E)$.

Bayes' rule follows by writing the same intersection two ways:

If $C_1,\dots,C_m$ are mutually exclusive hidden cases that cover the sample space, they form a partition. The denominator can be expanded by total probability:

for hidden cases with positive prior probability.

That gives the form used in classification, diagnosis, and Bayesian updating:

The numerator is likelihood times prior. The denominator is the total evidence. In the demo, the hidden cases are "coin A was chosen" and "coin B was chosen," while $O$ is the observed Head or Tail.