Video Generation

Video generation adds temporal modeling on top of image diffusion, treating a video as a high-dimensional tensor $x \in \mathbb{R}^{F \times H \times W \times 3}$ ($F$ RGB frames). Video diffusion keeps the DDPM noise-prediction objective $\mathbb{E}_{t,x,\epsilon}\left[\Vert \epsilon - \epsilon_\theta(x_t, t, c) \Vert^2\right]$; the difference lies in how the temporal axis is modeled:

1. 3D-conv U-Net (Video Diffusion Models): 2D convs become spatio-temporal 3D convs with shared weights across frames, implicitly modeling time, seamless with image pretraining but with limited temporal receptive field;

2. Factorized attention: spatial attention followed by temporal attention, cutting complexity from $O(F^2 H^2 W^2)$ to $O(FHW \cdot (FH + HW))$, a transitional design (ManyWorlds/VDM);

3. Causal spatio-temporal attention (Sora-style DiT): the mask forces frame $t$ to attend only to $t' \le t$ ($M_{i,j} = -\infty, \forall t_j > t_i$), explicitly causal and supporting arbitrary length/resolution at inference.

Cascade generation is standard: first generate a low-resolution backbone (latent frames $F/4 \times H/8 \times W/8$), then two spatio-temporal upsamplers enlarge it toward 1080p; each stage trains independently and runs in sequence, cutting memory and sampling cost drastically.

text-to-video (Sora / Kling / Veo / Runway Gen-3), image-to-video, video extension and editing; interviews probe the three temporal-modeling options and cascade roles.

video data is scarce, static/repetitive clips abound, and text-video alignment is noisy; cascade + causal modeling learns global motion at low resolution first, then high-frequency texture, mitigating flicker, motion drift and uncontrollable one-shot generation.

How do video diffusion models model time — tradeoffs among 3D conv, factorized attention, and causal attention?

How is the causal attention mask written, and what does it unlock?

What is cascade generation, and what does each spatio-temporal upsampler do?

Common video artifacts (flicker/drift/repetition) — causes and mitigations?

Key video training-data challenges compared with image data?