VAE & Reparameterization

A VAE is a variational generative model trained on the evidence lower bound: $\log p(x) \ge \mathbb{E}_{q(z|x)}[\log p(x|z)] - D_{KL}(q(z|x) \Vert p(z))$ — a reconstruction term plus a KL regularizer toward the prior. Because sampling from $q(z|x)$ gives an expectation non-differentiable w.r.t. parameters, the reparameterization trick writes $z = \mu + \sigma \odot \epsilon$ with $\epsilon \sim \mathcal{N}(0, I)$, moving the randomness into parameter-free noise so the sampling path is backprop-compatible.

generative modeling, latent representation learning, anomaly detection, and as the conceptual foundation of diffusion models (the noising process is a hierarchical VAE); interviews drill ELBO derivation and the reparameterization trick.

maximizing $\log p(x)$ directly requires an intractable marginal integral; VAEs turn inference into an SGD-optimizable lower bound. Unlike an autoencoder — a deterministic encoder with no prior constraint, giving an unstructured latent space that cannot be sampled for generation — the VAE pushes $q(z|x)$ toward the prior $p(z)$ and becomes generative; the cost is blurry samples (mode averaging: one $z$ must explain many modes, so outputs average over them) from the Gaussian pixel likelihood — the mirror image of GANs, which are sharp but hard to train.

Derive the VAE ELBO. Why is it a lower bound on $\log p(x)$?

What problem does the reparameterization trick solve? Why can't we sample straight from $q(z|x)$?

Core difference between VAE and AE? Why can't an AE generate new samples?

Why are VAE samples blurry? The mechanism of mode averaging?

How are the KL and reconstruction terms balanced? What are $\beta$-VAE and posterior collapse?