Continuous Function

Bayesian Inference

Bayesian inference updates a prior distribution over unknowns into a posterior by multiplying by the likelihood and normalizing.

Intermediate · Undergraduate mathematics · about 17 minutes

Reading map and next steps

Intuition

The idea in plain words, before the symbols.

Maximum likelihood asks: which parameter makes the data look most probable?

Bayesian inference asks a different question: after seeing the data, what should we believe about each possible parameter?

That difference matters when data is scarce or uncertainty matters. The likelihood is a score of parameters by data fit. A prior is what you believed before the data. A posterior is the updated distribution after both forces are combined.

For a coin with unknown head probability θ\theta, maximum likelihood might pick one number such as θ^=0.8\hat\theta=0.8. Bayesian inference keeps a whole distribution over plausible θ\theta values. With little data, the prior can still matter. With lots of data, and with a prior that gives nonzero density near the data-favored values, the likelihood usually concentrates the posterior near the parameters that explain the observations.

So Bayesian inference is not maximum likelihood with a correction added. It produces a different kind of answer: MLE returns one estimate, while Bayesian inference returns a distribution over the unknown quantity.

Study prompts for this section: Intuition
Study prompts

Each button copies a prompt about this section to your clipboard, for an AI assistant you already use. Nothing is sent by this site.

Mathematics

Definitions, assumptions and the derivation.

Equation 1Equation 2Equation 3

Let θ∈Θ\theta\in\Theta be an unknown parameter and let DD be observed data. Bayesian inference starts with a prior distribution or density p(θ)p(\theta) and a likelihood p(D∣θ)p(D\mid\theta). Bayes' rule gives the posterior:

p(θ∣D)=p(D∣θ)p(θ)p(D).p(\theta\mid D)=\frac{p(D\mid\theta)p(\theta)}{p(D)}.

For a continuous parameter, the denominator

p(D)=∫p(D∣θ)p(θ) dθp(D)=\int p(D\mid\theta)p(\theta)\,d\theta

is the evidence or marginal likelihood. Its job is to normalize the posterior so it integrates to 11.

If θ\theta lives in a discrete set, replace the integral with a sum. The job is the same: add up prior-weighted likelihood over all possible parameter values.

For parameter comparison, the key proportional form is

p(θ∣D)∝p(D∣θ)p(θ).p(\theta\mid D)\propto p(D\mid\theta)p(\theta).

This is the central contrast with maximum likelihood. The likelihood p(D∣θ)p(D\mid\theta) is a function of θ\theta, but it is not automatically a probability distribution over θ\theta. Multiplying by a prior and normalizing turns it into the posterior distribution.

For a coin with unknown head probability θ∈[0,1]\theta\in[0,1], suppose the data has hh heads and tt tails, with n=h+tn=h+t. The likelihood shape is

p(D∣θ)∝θh(1−θ)t.p(D\mid\theta)\propto \theta^h(1-\theta)^t.

If DD records only the count of heads, the full binomial likelihood also has a factor (nh)\binom{n}{h}. That factor does not depend on θ\theta, so it disappears in proportional calculations.

If the prior is a beta distribution with α,β>0\alpha,\beta>0,

θ∼Beta⁡(α,β),\theta\sim\operatorname{Beta}(\alpha,\beta),

then

p(θ)∝θα−1(1−θ)β−1.p(\theta)\propto \theta^{\alpha-1}(1-\theta)^{\beta-1}.

Multiplying prior and likelihood gives

p(θ∣D)∝θα+h−1(1−θ)β+t−1,p(\theta\mid D)\propto \theta^{\alpha+h-1}(1-\theta)^{\beta+t-1},

so the posterior is

θ∣D∼Beta⁡(α+h,β+t).\theta\mid D\sim\operatorname{Beta}(\alpha+h,\beta+t).

The maximum-likelihood estimate is

θ^MLE=hh+t\hat\theta_{\mathrm{MLE}}=\frac{h}{h+t}

when h+t>0h+t>0. The posterior mean is

E[θ∣D]=α+hα+β+h+t.\mathbb E[\theta\mid D]=\frac{\alpha+h}{\alpha+\beta+h+t}.

When n>0n>0, the posterior mean can also be written as

E[θ∣D]=α+βα+β+n⋅αα+β+nα+β+n⋅hn.\mathbb E[\theta\mid D]=\frac{\alpha+\beta}{\alpha+\beta+n}\cdot\frac{\alpha}{\alpha+\beta}+\frac{n}{\alpha+\beta+n}\cdot\frac{h}{n}.

This is the clean bridge to maximum likelihood: the posterior mean is a weighted compromise between the prior mean and the MLE. The posterior distribution is the richer object; the mean is only one summary of it.

It is safe to think of α+β\alpha+\beta as prior strength for this averaging formula. The density exponents are α−1\alpha-1 and β−1\beta-1, so pseudo-count language is only a mnemonic. The exact conjugate update is the parameter update Beta⁡(α,β)→Beta⁡(α+h,β+t)\operatorname{Beta}(\alpha,\beta)\to\operatorname{Beta}(\alpha+h,\beta+t).

Study prompts for this section: Mathematics
Study prompts

Each button copies a prompt about this section to your clipboard, for an AI assistant you already use. Nothing is sent by this site.

Code

The same calculation as runnable code, in the notation of the derivation.

Code example 1import math import numpy as np def log_beta_fn(a, b): return math.lgamma(a) + math.lgamma(b)...python
import math
import numpy as np

def log_beta_fn(a, b):
    return math.lgamma(a) + math.lgamma(b) - math.lgamma(a + b)

def log_beta_pdf(theta, a, b):
    if theta <= 0 or theta >= 1:
        return -math.inf
    return (a - 1) * math.log(theta) + (b - 1) * math.log(1 - theta) - log_beta_fn(a, b)

alpha, beta = 2.0, 2.0
heads, tails = 8, 2

post_alpha = alpha + heads
post_beta = beta + tails
n = heads + tails

mle = None if n == 0 else heads / n
prior_mean = alpha / (alpha + beta)
posterior_mean = post_alpha / (post_alpha + post_beta)

print("MLE:", "undefined" if mle is None else round(mle, 3))
print("prior mean:", round(prior_mean, 3))
print("posterior mean:", round(posterior_mean, 3))
print("posterior:", f"Beta({post_alpha:.1f}, {post_beta:.1f})")

# Grid approximation to show prior * likelihood -> posterior shape.
grid = np.linspace(0.01, 0.99, 99)
log_prior = np.array([log_beta_pdf(theta, alpha, beta) for theta in grid])
log_likelihood = heads * np.log(grid) + tails * np.log(1 - grid)
log_unnormalized_posterior = log_prior + log_likelihood

# Normalize on the grid for inspection.
weights = np.exp(log_unnormalized_posterior - log_unnormalized_posterior.max())
weights = weights / weights.sum()
grid_mean = float(np.sum(grid * weights))

print("grid posterior mean approx:", round(grid_mean, 3))

The code mirrors the math: the prior is a density over θ\theta, the likelihood scores the observed heads and tails for each θ\theta, and the posterior beta parameters add the observed counts to the prior parameters.

Study prompts for this section: Code
Study prompts

Each button copies a prompt about this section to your clipboard, for an AI assistant you already use. Nothing is sent by this site.

Demo

Change an input, predict the result, and see which quantities respond.

ExploreChange one control at a time and name what responds.An optional observation guide follows the demo.

Explore Bayesian Inference

Loading the interactive demo…

Observation guide

Optional and ungraded. Opening a guide saves nothing.

Choose what to inspect in Bayesian Inference, then open its guide.

Use the sliders and presets to compare prior strength against observed data. The prior and posterior curves are densities over θ\theta. The likelihood curve is normalized only for display so its shape can be compared on the same plot.

Try the "strong prior, little data" preset, then the "data wins" preset. The MLE only follows the observed fraction. The posterior mean moves between prior belief and data fit, and the full posterior curve shows how much uncertainty remains.

Before the posterior is revealed, the demo asks which of five outcomes the data will produce. The "balanced update" preset uses the code's numbers, α=β=2\alpha=\beta=2, h=8h=8, t=2t=2. The prior mean 0.50.5 gets weight 4/144/14 and the MLE 0.80.8 gets 10/1410/14, so E[θ∣D]=414⋅0.5+1014⋅0.8=1014≈0.714\mathbb E[\theta\mid D]=\tfrac{4}{14}\cdot0.5+\tfrac{10}{14}\cdot0.8=\tfrac{10}{14}\approx0.714. Subtracting the prior mean from the weighted form gives E[θ∣D]−αα+β=nα+β+n(hn−αα+β)\mathbb E[\theta\mid D]-\tfrac{\alpha}{\alpha+\beta}=\tfrac{n}{\alpha+\beta+n}\big(\tfrac{h}{n}-\tfrac{\alpha}{\alpha+\beta}\big): the mean moves the fraction n/(α+β+n)n/(\alpha+\beta+n) of the way to the MLE, less than half exactly when n<α+βn<\alpha+\beta, exactly half when n=α+βn=\alpha+\beta and more than half when n>α+βn>\alpha+\beta.

That fraction measures movement only when the two means differ. With no observations the posterior is the prior. When h/nh/n already equals the prior mean, the weighted average stays put whatever the weights, yet the posterior is still Beta⁡(α+h,β+t)\operatorname{Beta}(\alpha+h,\beta+t) rather than Beta⁡(α,β)\operatorname{Beta}(\alpha,\beta). A beta distribution with mean μ\mu has variance μ(1−μ)/(α+β+1)\mu(1-\mu)/(\alpha+\beta+1), so with the mean fixed the added counts strictly reduce the spread. The demo decides the outcome by exact comparisons of these counts; its displayed decimals are rounded.

Study prompts for this section: Demo
Study prompts

Each button copies a prompt about this section to your clipboard, for an AI assistant you already use. Nothing is sent by this site.

In focus

Concept: Bayesian Inference

What is the smallest example of Bayesian Inference you can work through by hand, and what does it show?

BeforeDistributionsNextCross-Entropy
DetailsProbability
About this itemProbability
3 sources citedChecking this browser
Reference idconcept:probability/bayesian-inference
Math3 equationsp(\theta\mid D)=(p(D\mid\theta)p(\theta))/(p(D)).CodeBayesian Inference code example 1if theta <= 0 or theta >= 1:DemoBayesian Inference interactive demoChange one control at a time and name what responds.

After the first pass

Go deeper

Optional material for a second reading. Sources are always shown; the other panels open from their headings or from these links, and closing one keeps what you wrote or revealed.
What to look for in each sectionChoose a section and what you expect to see in it, then read short advice.

Section by section

Choose what to look for in each section

Bayesian inference updates a prior distribution over unknowns into a posterior by multiplying by the likelihood and normalizing.

Choose what to look for (no demo yet)01 / Intuition
Illustration of prior and likelihood curves combining into a sharper posterior belief.
What to look for

Start with the picture, metaphor, or geometric mechanism.

Choose first

Choose what you expect to see in this section of Bayesian Inference.

Questions to ask of the pictureChoose a part of the picture and what you expect, then read short advice.

Looking at the picture

Questions to ask of the picture

Bayesian inference updates a prior distribution over unknowns into a posterior by multiplying by the likelihood and normalizing.

4 of 4 sections writtenNo live demo yet
Question

Which part of the picture should you look at first?

Choose first

Pick the part of the picture you expect to explain Bayesian Inference.

Sources

References for this page

A reference shows where an idea comes from; it does not vouch for every step on this page.
book · 2020Mathematics for Machine LearningDeisenroth, Faisal, and Ong
Why it is cited

Grounds the probability notation and Bayes-rule prerequisites needed for the page.

Open source
book · 2022Probabilistic Machine Learning: An IntroductionMurphy
Why it is cited

Grounds Bayesian inference as posterior updating under probabilistic models.

Open source
paper · 1763An Essay towards solving a Problem in the Doctrine of ChancesBayes
Why it is cited

Philosophical Transactions of the Royal Society of London 53, 370–418 (1763, published posthumously by Richard Price). Infers an unknown chance of success from observed successes and failures, starting from a uniform prior.

Open source
What each claim rests on1 claim, each with its sources, where it appears on this page, and its caveats.

Claims and sources

Bayesian inference updates a prior distribution over unknowns into a posterior by multiplying by the likelihood and normalizing.

Checked against the cited passages

The checks were made by this site, not by an independent reviewer.

Bayesian inference updates a prior distribution over unknown parameters by multiplying it with the likelihood of observed data and normalizing by evidence, producing a posterior distribution rather than a single maximum-likelihood estimate.
Sources
Mathematics for Machine LearningGrounds the probability notation and Bayes-rule prerequisites needed for the page.Probabilistic Machine Learning: An IntroductionGrounds Bayesian inference as posterior updating under probabilistic models.
What the sources support

Deisenroth et al. ground Bayes' rule as posterior = likelihood times prior divided by evidence and define evidence as the posterior normalizer; Murphy grounds Bayesian parameter inference as updating p(theta...

Where to see it on this page
Mathematics for Machine LearningProbabilistic Machine Learning: An IntroductionEquation 1Equation 2Code example 1Demo
Equation 1
p(θ∣D)=p(D∣θ)p(θ)p(D).p(\theta\mid D)=\frac{p(D\mid\theta)p(\theta)}{p(D)}.
Equation 2
p(D)=∫p(D∣θ)p(θ) dθp(D)=\int p(D\mid\theta)p(\theta)\,d\theta
Caveat

Covers posterior updating and the contrast with MLE, not approximate inference, hierarchical models, asymptotics or decisions. The code and demo use the beta-Bernoulli model; the binomial coefficient they om...

StatusChecked against the cited passages.

MML supports Bayes-rule notation with prior, likelihood, evidence, and posterior labels, defines evidence/marginal likelihood as the normalizer, and notes that the full posterior retains more information than focusing on a maximum/statistic. Murphy directly supports parameter-level Bayesian updating as p(theta|D) proportional to p(theta)p(D|theta) normalized by p(D), contrasts this with MLE as a single likelihood-maximizing parameter estimate, and gives the beta-Bernoulli/binomial posterior update used by the page.

Explain it without the pageWrite your own explanation before asking for feedback. Your draft and any hints stay while this page is open.

Practice · Bayesian Inference

Try the idea in your own words

Bayesian inference updates a prior distribution over unknowns into a posterior by multiplying by the likelihood and normalizing.

Concept · Selected for practice

Bayesian Inference

What it rests on: Sources: Mathematics for Machine Learning; Probabilistic Machine Learning: An Introduction; An Essay towards solving a Problem in the Doctrine of C...

Context and links

Probability

Choose a task

Explain the mechanism

For Bayesian Inference: What is the smallest example of Bayesian Inference you can work through by hand, and what does it show? Explain your answer, including what changes, why, and which assumption matters.

No answer yet

A rough first thought is enough. Your draft stays when you change tasks.

Your draft stays on this page and is cleared when you leave or reload.

A little help · Explain

Open one hint at a time. These are suggestions, not your answer or a grade.

0 of 3 hints shown for this question.

    Clearing your answer keeps this help record. Outside help is your own report; this page cannot check it.

    Where am I stuck? (optional)
    Your own description, not an automatic diagnosis

    Choose one, or leave this unspecified. Select it again to clear it.

    Take your draft to a feedback conversation

    No AI feedback runs here. You can copy a prompt to use elsewhere; nothing is sent automatically. Review the text before sharing, and leave out private information.

    Write an attempt before copying a feedback prompt.

    A draft, or an AI reply to it, is not a test of what you have learned. To check that, try a different case later without help.

    Ask about this pagePick one item on this page and copy a prompt about it into an AI assistant you already use. Notes you write here stay in this browser.
    Ask about this pageClose
    ConceptBayesian InferenceSources: Mathematics for Machine Learning; Probabilistic Machine Learning: An Introduction; An Essay towards solving a Problem in the Doctrine of C...if theta <= 0 or theta >= 1:

    Your question

    Choose what your question is about

    Pick the idea, equation, source, code, claim, misconception or demo state you want to ask about. The prompt you copy includes it, so the answer can stay on that item.
    Next local actionNo local draft saved yet

    Open the draft below to save one note and next action in this browser.

    conceptProbability

    Bayesian Inference

    Starting question

    What is the smallest example of Bayesian Inference you can work through by hand, and what does it show?

    SourcesCheck the 3 cited sources listed under Sources.Notes can be saved for this item
    Points of view on this item

    Fixed prompts generated from this item. They are not comments from people or independent reviews.

    Learner’s first stepAsk what would make "Bayesian Inference" feel predictable rather than familiar.
    Assumption

    The 3 cited sources must support this exact item, not just the surrounding topic.

    Source-checking summary

    Connect the definition to one equation, piece of code or demo before widening the discussion.

    Proposed experiment

    Change one input, then check that the same relationship holds in the mathematics, the code and the demo.

    Next action

    You can state the mechanism in your own words

    Evidence4 checks
    ObservationChecking for a saved observation
    ActionReady for one action
    PromptLearner prompt ready to copy
    Go to this item
    01ObservationChecking what this browser has saved
    02EvidenceChecking for a saved observation
    03SourcesCheck the 3 cited sources listed under Sources.
    04Next stepSave one next action
    Local action draftNo local draft saved yetOpen when you are ready to write one next action
    Local action draft

    This draft stays in this browser and is attached to this item only.

    No local draft saved.
    Evidence to inspect
    • What the 3 cited sources say about this exact item
    • The definition, its prerequisites, and a contrasting concept
    • The equation or code that makes the concept concrete
    • One demo state that shows what stays the same as the inputs change
    What would resolve this
    • You can state the mechanism in your own words
    • You can name the prerequisite that would clear up the confusion
    • You can predict how the result changes when one input changes
    Reference idconcept/concept-notebook/probability/bayesian-inference concept:probability/bayesian-inference