DiT & 3D Causal VAE

DiT (Diffusion Transformer) replaces the U-Net with a pure Transformer as the diffusion backbone — the underlying architecture of Sora, SD3 and FLUX. Two core designs:

Patchify: the latent map $z \in \mathbb{R}^{h \times w \times c}$ is cut into $p \times p$ patches and linearly embedded into $T = \frac{h}{p} \cdot \frac{w}{p}$ tokens of dimension $d$ (video adds 3D spatio-temporal positional encoding). Larger patches mean fewer tokens and cheaper compute at the cost of spatial detail (typically $p = 2$).

AdaLN conditioning: timestep $t$ and condition $c$ are not concatenated into tokens; an MLP maps them to per-token scale/shift that modulate LayerNorm:

Sora uses AdaLN-Zero: the MLP output is zero-initialized and the modulation sits before the residual branch, so the model starts as an identity mapping — more stable and faster to converge, with fewer parameters and less compute than cross-attention injection.

Sora 3D causal VAE: video is compressed into latent space before the DiT — a causal 8×8 spatial, 4× temporal compression of $F \times H \times W \times 3$ into $F/4 \times H/8 \times W/8 \times 16$, a $8 \times 8 \times 4 = 256\times$ volume reduction. Causal (current frame and earlier only) means fixed-length blocks at training but frame-wise or variable-length streaming at inference — any duration shares one latent space.

text-to-video backbones (Sora), flagship text-to-image (SD3/FLUX MMDiT); interviews ask for token-count math of patchify and why AdaLN-Zero stabilizes training.

U-Net's local inductive bias under-models long-range temporal dependencies and scales poorly with data; DiT provides a scalable uniform Transformer, and the 3D VAE's 256× compression keeps minute-long, high-res video tractable — the enabling combination behind Sora.

Test Your Knowledge: Practice Questions for "DiT & 3D Causal VAE"

Single choice pitfall questions with instant feedback and mistake tracking.