Latent Diffusion (LDM)

Latent Diffusion Models (LDM) run the DDPM process in a VAE latent space instead of pixel space, then decode back. An image $x \in \mathbb{R}^{H \times W \times 3}$ is compressed by the VAE encoder into $z \in \mathbb{R}^{h \times w \times c}$; Stable Diffusion uses $8\times$ spatial downsampling: a $512\times 512$ image becomes a $64\times 64\times 4$ latent ($c=4$). Denoising happens entirely on $z$:

The condition $c$ (e.g. CLIP text embeddings) is injected into the U-Net/DiT middle layers via cross-attention; after sampling, $z_0$ is mapped back by $\mathrm{VAE}_{dec}$. The full training loss combines three terms: latent diffusion reconstruction + perceptual loss (LPIPS) + adversarial/GAN loss, with mild KL regularization, balancing pixel fidelity and perceptual quality. Architecture evolution: SD1/2 use U-Net (self-attention + cross-attention + timestep embedding); SD3/Flux use MMDiT (a mixed text-image DiT with patchify and AdaLN-zero for timestep/text conditioning).

the backbone of the Stable Diffusion family and most text-to-image/text-to-video models; interview favorites — “why diffuse in latent space instead of pixel space?” and “how much compute does 8× downsampling save?”.

pixel-space diffusion is computationally intractable above $256\times256$ — attention and feature maps grow quadratically with resolution; the $8\times$ spatial reduction cuts feature-map area to $1/64$, lowering training/sampling compute by roughly two orders of magnitude, while the decoder restores visual detail, the encoder absorbs semantic compression, and conditions (text, ControlNet) plug into the latent space uniformly.

Why does Stable Diffusion diffuse in VAE latent space instead of pixel space; how much compute does 8× downsampling save?

Which terms make up the LDM training loss, and why add perceptual and adversarial losses?

How is text conditioning injected into the latent diffusion network; where does cross-attention sit in U-Net/DiT?

Key structural differences between SD1/2 U-Net and SD3/Flux MMDiT?

Tradeoffs of latent vs pixel-space diffusion in training speed, memory and image quality?