VAE & Reparameterization

A VAE is a variational generative model trained on the evidence lower bound: $\log p(x) \ge \mathbb{E}_{q(z|x)}[\log p(x|z)] - D_{KL}(q(z|x) \Vert p(z))$ — a reconstruction term plus a KL regularizer toward the prior. Because sampling from $q(z|x)$ gives an expectation non-differentiable w.r.t. parameters, the reparameterization trick writes $z = \mu + \sigma \odot \epsilon$ with $\epsilon \sim \mathcal{N}(0, I)$, moving the randomness into parameter-free noise so the sampling path is backprop-compatible.

generative modeling, latent representation learning, anomaly detection, and as the conceptual foundation of diffusion models (the noising process is a hierarchical VAE); interviews drill ELBO derivation and the reparameterization trick.

maximizing $\log p(x)$ directly requires an intractable marginal integral; VAEs turn inference into an SGD-optimizable lower bound. Unlike an autoencoder — a deterministic encoder with no prior constraint, giving an unstructured latent space that cannot be sampled for generation — the VAE pushes $q(z|x)$ toward the prior $p(z)$ and becomes generative; the cost is blurry samples (mode averaging: one $z$ must explain many modes, so outputs average over them) from the Gaussian pixel likelihood — the mirror image of GANs, which are sharp but hard to train.

Test Your Knowledge: Practice Questions for "VAE & Reparameterization"

Single choice pitfall questions with instant feedback and mistake tracking.