GMM Soft Assignment

A Gaussian mixture model (GMM) models the data distribution as a weighted sum of $K$ Gaussian components, $p(x) = \sum_{k=1}^{K} \pi_k \mathcal{N}(x \mid \mu_k, \Sigma_k)$, with mixing coefficients satisfying $\sum_k \pi_k = 1, \ \pi_k \ge 0$.Parameters are estimated with EM: the E-step computes responsibilities (posteriors, i.e. soft assignments) $\gamma_{ik} = \frac{\pi_k \mathcal{N}(x_i \mid \mu_k, \Sigma_k)}{\sum_{j=1}^{K} \pi_j \mathcal{N}(x_i \mid \mu_j, \Sigma_j)}$; the M-step re-estimates $N_k = \sum_i \gamma_{ik}$, $\pi_k = \tfrac{N_k}{N}$, $\mu_k = \tfrac{1}{N_k} \sum_i \gamma_{ik} x_i$, and $\Sigma_k = \tfrac{1}{N_k} \sum_i \gamma_{ik} (x_i - \mu_k)(x_i - \mu_k)^T$.As $\Sigma_k = \sigma^2 I$ with $\sigma \to 0$, responsibilities collapse to hard 0/1 assignments and the EM update reduces to cluster means — K-Means is exactly the zero-variance EM special case of GMM.

soft clustering and density estimation, anomaly detection (low-likelihood samples), speaker recognition, background modeling, and as an upgrade when K-Means underperforms; interviews ask for the responsibility formula and the K-Means-as-GMM derivation.

K-Means only produces hard clusters and assumes spherical shapes; GMM outputs a posterior probability for every sample (confidence and uncertainty), and per-component covariances fit elliptical clusters of varying scale and orientation. The cost: far more parameters — $d + \tfrac{d(d+1)}{2}$ covariance parameters per Gaussian — a non-convex likelihood that needs K-Means++ initialization and constrained covariances (diagonal/shared/regularized) to avoid singular degeneracy.

Write the GMM density $p(x) = \sum_k \pi_k \mathcal{N}(x \mid \mu_k, \Sigma_k)$ and the responsibility $\gamma_{ik}$; why does $\sum_k \gamma_{ik} = 1$?

Why is K-Means the EM special case of GMM as $\Sigma_k \to 0$? How does the responsibility collapse to hard 0/1 assignment?

Derive the GMM M-step updates $\pi_k = \tfrac{N_k}{N}$ and $\mu_k = \tfrac{1}{N_k} \sum_i \gamma_{ik} x_i$; why are these weighted means?

When do GMM and K-Means clusters differ significantly? What extra value does soft assignment bring (confidence, anomaly detection)?

Parameter count and overfitting risk of GMM? Why can covariances degenerate to singular solutions, and what constraints help?