Latent Diffusion (LDM)

Latent Diffusion Models (LDM) run the DDPM process in a VAE latent space instead of pixel space, then decode back. An image $x \in \mathbb{R}^{H \times W \times 3}$ is compressed by the VAE encoder into $z \in \mathbb{R}^{h \times w \times c}$; Stable Diffusion uses $8\times$ spatial downsampling: a $512\times 512$ image becomes a $64\times 64\times 4$ latent ($c=4$). Denoising happens entirely on $z$:

The condition $c$ (e.g. CLIP text embeddings) is injected into the U-Net/DiT middle layers via cross-attention; after sampling, $z_0$ is mapped back by $\mathrm{VAE}_{dec}$. The full training loss combines three terms: latent diffusion reconstruction + perceptual loss (LPIPS) + adversarial/GAN loss, with mild KL regularization, balancing pixel fidelity and perceptual quality. Architecture evolution: SD1/2 use U-Net (self-attention + cross-attention + timestep embedding); SD3/Flux use MMDiT (a mixed text-image DiT with patchify and AdaLN-zero for timestep/text conditioning).

the backbone of the Stable Diffusion family and most text-to-image/text-to-video models; interview favorites — “why diffuse in latent space instead of pixel space?” and “how much compute does 8× downsampling save?”.

pixel-space diffusion is computationally intractable above $256\times256$ — attention and feature maps grow quadratically with resolution; the $8\times$ spatial reduction cuts feature-map area to $1/64$, lowering training/sampling compute by roughly two orders of magnitude, while the decoder restores visual detail, the encoder absorbs semantic compression, and conditions (text, ControlNet) plug into the latent space uniformly.

Test Your Knowledge: Practice Questions for "Latent Diffusion (LDM)"

Single choice pitfall questions with instant feedback and mistake tracking.