Variational Inference & ELBO

Variational inference approximates an intractable posterior $p(z|x)$ with a tractable distribution $q(z|x)$ via the evidence lower bound (ELBO). Exact decomposition of the log-evidence: $\log p(x) = \text{ELBO}(q) + D_{KL}(q(z|x) \Vert p(z|x))$, where $\text{ELBO}(q) = \mathbb{E}_{q(z|x)}\left[\log\frac{p(x,z)}{q(z|x)}\right] = \mathbb{E}_q[\log p(x,z)] + H(q) = \mathbb{E}_q[\log p(x|z)] - D_{KL}(q(z|x) \Vert p(z))$. The bound is constructed via Jensen: since $\log$ is concave, $\log p(x) = \log\mathbb{E}_q\left[\frac{p(x,z)}{q(z|x)}\right] \ge \mathbb{E}_q\left[\log\frac{p(x,z)}{q(z|x)}\right] = \text{ELBO}(q)$; expanding $\mathbb{E}_q[\log p(x)] = \text{ELBO} + \mathbb{E}_q\left[\log\frac{q}{p(z|x)}\right]$ yields the exact decomposition, with equality iff $q = p(z|x)$, where the KL gap is 0 and ELBO reaches $\log p(x)$.

the training objective of VAEs (reconstruction + KL regularization with the reparameterization trick), Bayesian neural networks, LDA topic modeling; "derive the ELBO" and "why does a VAE optimize the ELBO" are canonical interview questions.

the posterior $p(z|x)$ is usually not analytically integrable and MCMC is infeasible at scale; variational inference turns inference into optimization, compatible with SGD, and yields an approximate posterior with uncertainty. The cost: restricted approximation families (e.g. mean-field independence) tend to underestimate posterior variance.

Test Your Knowledge: Practice Questions for "Variational Inference & ELBO"

Single choice pitfall questions with instant feedback and mistake tracking.