Probability Basics

For this page, start with a finite probability model. A probability space is $(\Omega,\mathcal F,P)$:

• $\Omega$: sample space (all outcomes)
• $\mathcal{F}$: events (in a finite model, usually all subsets of $\Omega$; in general, a sigma-algebra of measurable subsets)
• $P$: probability measure

For events $E\in\mathcal F$, the basic axioms are:

1. $P(E) \ge 0$
2. $P(\Omega)=1$
3. If $E_1,E_2,\dots$ are pairwise disjoint, then

$$P\left(\bigcup_i E_i\right)=\sum_i P(E_i).$$

For finite examples, this reduces to adding the probabilities of disjoint pieces.

The complement rule follows:

$$P(E^c)=1-P(E).$$

Conditional probability means updating probabilities after observing an event $O$:

$$P(E\mid O) = \frac{P(E\cap O)}{P(O)} \quad (P(O)>0).$$

The denominator matters. Conditioning does not merely keep $E\cap O$; it rescales all probability inside $O$ so the new total mass is $1$. If $P(O)=0$, there is no mass inside the observed event to renormalize, so $P(E\mid O)$ is undefined.

The product rule is the same equation rearranged:

$$P(E\cap O)=P(E\mid O)P(O)=P(O\mid E)P(E),$$

when the conditioning events have positive probability.

Independence means learning one event does not change the probability of the other:

$$E \perp O \iff P(E\cap O)=P(E)P(O).$$

Equivalently, when $P(O)>0$, independence means $P(E\mid O)=P(E)$.

Bayes' rule follows by writing the same intersection two ways:

$$P(E\mid O) = \frac{P(O\mid E)P(E)}{P(O)}.$$

If $C_1,\dots,C_m$ are mutually exclusive hidden cases that cover the sample space, they form a partition. The denominator can be expanded by total probability:

$$P(O)=\sum_{j=1}^m P(O\mid C_j)P(C_j),$$

for hidden cases with positive prior probability.

That gives the form used in classification, diagnosis, and Bayesian updating:

$$P(C_i\mid O)=\frac{P(O\mid C_i)P(C_i)}{\sum_{j=1}^m P(O\mid C_j)P(C_j)}.$$

The numerator is likelihood times prior. The denominator is the total evidence. In the demo, the hidden cases are "coin A was chosen" and "coin B was chosen," while $O$ is the observed Head or Tail.