Variational Inference & ELBO

Variational inference approximates an intractable posterior $p(z|x)$ with a tractable distribution $q(z|x)$ via the evidence lower bound (ELBO). Exact decomposition of the log-evidence: $\log p(x) = \text{ELBO}(q) + D_{KL}(q(z|x) \Vert p(z|x))$, where $\text{ELBO}(q) = \mathbb{E}_{q(z|x)}\left[\log\frac{p(x,z)}{q(z|x)}\right] = \mathbb{E}_q[\log p(x,z)] + H(q) = \mathbb{E}_q[\log p(x|z)] - D_{KL}(q(z|x) \Vert p(z))$. The bound is constructed via Jensen: since $\log$ is concave, $\log p(x) = \log\mathbb{E}_q\left[\frac{p(x,z)}{q(z|x)}\right] \ge \mathbb{E}_q\left[\log\frac{p(x,z)}{q(z|x)}\right] = \text{ELBO}(q)$; expanding $\mathbb{E}_q[\log p(x)] = \text{ELBO} + \mathbb{E}_q\left[\log\frac{q}{p(z|x)}\right]$ yields the exact decomposition, with equality iff $q = p(z|x)$, where the KL gap is 0 and ELBO reaches $\log p(x)$.

the training objective of VAEs (reconstruction + KL regularization with the reparameterization trick), Bayesian neural networks, LDA topic modeling; "derive the ELBO" and "why does a VAE optimize the ELBO" are canonical interview questions.

the posterior $p(z|x)$ is usually not analytically integrable and MCMC is infeasible at scale; variational inference turns inference into optimization, compatible with SGD, and yields an approximate posterior with uncertainty. The cost: restricted approximation families (e.g. mean-field independence) tend to underestimate posterior variance.

Fully derive $\log p(x) = \text{ELBO} + D_{KL}(q \Vert p)$?

How does Jensen's inequality prove that ELBO is a lower bound on $\log p(x)$?

Two equivalent ELBO forms (reconstruction + KL penalty)? Why does maximizing ELBO minimize the KL gap?

Relation between variational inference and EM? What do the E and M steps do?

What is the mean-field assumption? Why does VI tend to underestimate posterior variance?