Bayesian Updating and Conjugacy

Let $\theta$ be an unknown parameter and $D=\{x_1,\ldots,x_n\}$ be observed data. Bayes' rule gives

$$p(\theta\mid D) = \frac{p(D\mid\theta)p(\theta)}{p(D)}.$$

A prior family $\mathcal P$ is conjugate to a likelihood family when the posterior stays in $\mathcal P$ for every dataset in the model:

$$p(\theta)\in\mathcal P \quad\Rightarrow\quad p(\theta\mid D)\in\mathcal P.$$

The easiest exact example is the beta-Bernoulli model. Let $\theta\in[0,1]$ be the probability of success. The prior is

$$\theta \sim \operatorname{Beta}(\alpha,\beta), \qquad p(\theta)\propto \theta^{\alpha-1}(1-\theta)^{\beta-1}.$$

If the data has $h$ successes and $t$ failures, the Bernoulli likelihood has shape

$$p(D\mid\theta)\propto \theta^h(1-\theta)^t.$$

Multiplying likelihood and prior gives

$$p(\theta\mid D) \propto \theta^{\alpha+h-1}(1-\theta)^{\beta+t-1},$$

so

$$\theta\mid D \sim \operatorname{Beta}(\alpha+h,\beta+t).$$

The posterior predictive probability of the next success is the posterior mean:

$$p(x_{n+1}=1\mid D) = \mathbb E[\theta\mid D] = \frac{\alpha+h}{\alpha+\beta+h+t}.$$

For a $K$-class categorical model, let $\pi=(\pi_1,\ldots,\pi_K)$ be class probabilities and let the prior be

$$\pi \sim \operatorname{Dirichlet}(\alpha_1,\ldots,\alpha_K).$$

If $c_k$ is the count of class $k$, the likelihood contributes

$$\prod_{k=1}^{K}\pi_k^{c_k}.$$

The posterior is

$$\pi\mid D \sim \operatorname{Dirichlet}(\alpha_1+c_1,\ldots,\alpha_K+c_K),$$

and the posterior predictive probability for class $k$ is

$$p(x_{n+1}=k\mid D) = \frac{\alpha_k+c_k}{\sum_{j=1}^{K}\alpha_j+n}.$$

For a normal mean with known observation variance, the sufficient statistic is different. Let

$$x_i\mid \mu \sim \mathcal N(\mu,\sigma^2)$$

with known $\sigma^2$, and let the prior be

$$\mu \sim \mathcal N(\mu_0,\tau_0^{-1}),$$

where $\tau_0$ is prior precision. If $\bar x$ is the sample mean, the data precision is

$$\tau_D=\frac{n}{\sigma^2}.$$

The posterior is another normal distribution,

$$\mu\mid D\sim \mathcal N(\mu_n,\tau_n^{-1}),$$

with

$$\tau_n=\tau_0+\tau_D$$

and

$$\mu_n = \frac{\tau_0\mu_0+\tau_D\bar x}{\tau_0+\tau_D}.$$

So the update is still ledger arithmetic, but the ledger is precision-weighted rather than count-only.

The general exponential-family version explains why this keeps happening. Suppose the likelihood can be written as

$$p(x\mid \eta) = h(x)\exp\left(\eta^\top T(x)-A(\eta)\right),$$

where $\eta$ is a natural parameter, $T(x)$ is a sufficient statistic, and $A(\eta)$ is the log normalizer. A conjugate prior often has the form

$$p(\eta\mid \chi,\nu) \propto \exp\left(\eta^\top\chi-\nu A(\eta)\right).$$

For iid data, the likelihood contributes $\sum_i T(x_i)$ and $nA(\eta)$. The posterior hyperparameters become

$$\chi'=\chi+\sum_{i=1}^{n}T(x_i), \qquad \nu'=\nu+n.$$

This is the reusable invariant: conjugacy works when the prior is shaped to receive the likelihood's sufficient statistics.