Gaussian Mixture Models and EM

Let each data point be $x_i\in\mathbb R^d$. A $K$-component Gaussian mixture model defines the density

$$p_\theta(x_i) = \sum_{k=1}^K \pi_k\, \mathcal N(x_i\mid \mu_k,\Sigma_k),$$

where:

• $\pi_k\ge 0$ is the mixture weight for component $k$;
• $\sum_{k=1}^K \pi_k=1$;
• $\mu_k\in\mathbb R^d$ is the component mean;
• $\Sigma_k\in\mathbb R^{d\times d}$ is the component covariance matrix.

The observed-data log likelihood is

$$\ell(\theta) = \sum_{i=1}^n \log \sum_{k=1}^K \pi_k\, \mathcal N(x_i\mid \mu_k,\Sigma_k).$$

The log is outside a sum over hidden components, which is why the optimization is harder than fitting one Gaussian with known labels.

Introduce a hidden component label

$$z_i\in\{1,\ldots,K\}.$$

If $z_i$ were observed, we could count how many points belonged to each component and fit weighted Gaussian parameters. Since $z_i$ is hidden, EM computes the posterior probability of each possible label.

E-step

Given the current parameters $\theta^{old}$, define the responsibility

$$r_{ik} = p(z_i=k\mid x_i,\theta^{old}) = \frac{ \pi_k^{old}\mathcal N(x_i\mid \mu_k^{old},\Sigma_k^{old}) }{ \sum_{j=1}^K \pi_j^{old}\mathcal N(x_i\mid \mu_j^{old},\Sigma_j^{old}) }.$$

For each point $i$, the responsibilities sum to one:

$$\sum_{k=1}^K r_{ik}=1.$$

These are soft assignments. In k-means, a point contributes either $1$ or $0$ to a cluster. In a GMM, a point can contribute $0.73$ to one component and $0.27$ to another.

M-step

Define the effective soft count

$$N_k=\sum_{i=1}^n r_{ik}.$$

The updated mixture weight is

$$\pi_k^{new} = \frac{N_k}{n}.$$

The updated mean is the responsibility-weighted average:

$$\mu_k^{new} = \frac{1}{N_k} \sum_{i=1}^n r_{ik}x_i.$$

The updated covariance is the responsibility-weighted scatter:

$$\Sigma_k^{new} = \frac{1}{N_k} \sum_{i=1}^n r_{ik} (x_i-\mu_k^{new})(x_i-\mu_k^{new})^\top.$$

The interactive demo uses diagonal covariances, so it tracks only the per-coordinate variances. The full formula above is the object being simplified.

What EM Guarantees

Exact EM can be understood as coordinate ascent on a lower bound to the observed-data log likelihood. The E-step makes the bound tight at the current parameters by using the posterior over hidden labels. The M-step increases that bound by refitting parameters to the soft counts. Therefore one exact EM iteration does not decrease the observed-data log likelihood.

This is a local guarantee, not a global one. Mixture likelihoods can have multiple local optima, component labels are interchangeable, and covariance estimates can become ill-conditioned without constraints or regularization.