Bayesian Inference

Let $\theta\in\Theta$ be an unknown parameter and let $D$ be observed data. Bayesian inference starts with a prior distribution or density $p(\theta)$ and a likelihood $p(D\mid\theta)$. Bayes' rule gives the posterior:

$$p(\theta\mid D)=\frac{p(D\mid\theta)p(\theta)}{p(D)}.$$

For a continuous parameter, the denominator

$$p(D)=\int p(D\mid\theta)p(\theta)\,d\theta$$

is the evidence or marginal likelihood. Its job is to normalize the posterior so it integrates to $1$.

If $\theta$ lives in a discrete set, replace the integral with a sum. The job is the same: add up prior-weighted likelihood over all possible parameter values.

For parameter comparison, the key proportional form is

$$p(\theta\mid D)\propto p(D\mid\theta)p(\theta).$$

This is the central contrast with maximum likelihood. The likelihood $p(D\mid\theta)$ is a function of $\theta$, but it is not automatically a probability distribution over $\theta$. Multiplying by a prior and normalizing turns it into the posterior distribution.

For a coin with unknown head probability $\theta\in[0,1]$, suppose the data has $h$ heads and $t$ tails, with $n=h+t$. The likelihood shape is

$$p(D\mid\theta)\propto \theta^h(1-\theta)^t.$$

If $D$ records only the count of heads, the full binomial likelihood also has a factor $\binom{n}{h}$. That factor does not depend on $\theta$, so it disappears in proportional calculations.

If the prior is a beta distribution with $\alpha,\beta>0$,

$$\theta\sim\operatorname{Beta}(\alpha,\beta),$$

then

$$p(\theta)\propto \theta^{\alpha-1}(1-\theta)^{\beta-1}.$$

Multiplying prior and likelihood gives

$$p(\theta\mid D)\propto \theta^{\alpha+h-1}(1-\theta)^{\beta+t-1},$$

so the posterior is

$$\theta\mid D\sim\operatorname{Beta}(\alpha+h,\beta+t).$$

The maximum-likelihood estimate is

$$\hat\theta_{\mathrm{MLE}}=\frac{h}{h+t}$$

when $h+t>0$. The posterior mean is

$$\mathbb E[\theta\mid D]=\frac{\alpha+h}{\alpha+\beta+h+t}.$$

When $n>0$, the posterior mean can also be written as

$$\mathbb E[\theta\mid D]=\frac{\alpha+\beta}{\alpha+\beta+n}\cdot\frac{\alpha}{\alpha+\beta}+\frac{n}{\alpha+\beta+n}\cdot\frac{h}{n}.$$

This is the clean bridge to maximum likelihood: the posterior mean is a weighted compromise between the prior mean and the MLE. The posterior distribution is the richer object; the mean is only one summary of it.

It is safe to think of $\alpha+\beta$ as prior strength for this averaging formula. The density exponents are $\alpha-1$ and $\beta-1$, so pseudo-count language is only a mnemonic. The exact conjugate update is the parameter update $\operatorname{Beta}(\alpha,\beta)\to\operatorname{Beta}(\alpha+h,\beta+t)$.