M2-055M2: Classical Machine LearningClustering AlgorithmsMedium
⚡Flashcard Drill➔Mastery:
Derive the Expectation-Maximization (EM) Algorithm and Prove Its Monotonic Likelihood Ascent via Jensen's Inequality
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
The EM algorithm maximizes incomplete log-likelihood with latent variables by iteratively constructing a tight Evidence Lower Bound (ELBO) in the E-step via and maximizing that lower bound in the M-step.
📌 Key Takeaways
- •Latent Variable Log-Likelihood: ; intractable sum inside the logarithm.
- •E-step (Expectation): Computes the posterior distribution of latent variables: , making the ELBO touch the true likelihood.
- •M-step (Maximization): Updates parameters by maximizing expected complete log-likelihood: .
- •Monotonic Guarantee: ; guaranteed never to decrease likelihood.
📐 Mathematical Derivations
Proof of monotonic ascent via Jensen's Inequality: For any arbitrary latent distribution : . Decomposing the difference: . Thus, . In the E-step, setting drives , making . In the M-step, we choose . Therefore: , proving monotonic convergence.
🏭 Production Trade-offs
Classic instances of the EM algorithm in ML: (1) Gaussian Mixture Models (GMM): Latent variables ; E-step computes responsibilities , M-step updates . (2) Hidden Markov Models (Baum-Welch): Latent sequence states. (3) Variational Autoencoders (VAE): Amortized variational EM optimizing continuous latents.
⚠️ Common Interview Pitfalls
- ✕Assuming EM converges to the global maximum (it is a local ascent algorithm that can get trapped in local maxima or saddle points).
- ✕Singularities in GMM: If a Gaussian component collapses onto a single data point, variance , causing likelihood to diverge to (must constrain covariance eigenvalues).
🎯 Interviewer Follow-up Inquiries
- ?How does the Evidence Lower Bound (ELBO) in Variational Inference generalize the EM algorithm?
- ?Why do GMM likelihood surfaces have unbounded singularities when covariance matrices approach zero?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚