Progressive Video Training

Text-to-video training follows a three-stage progressive recipe (Sora, Stable Video Diffusion, Kling and others), gradually introducing the video distribution:

1. Image pretraining: train on massive image-text pairs (or load a T2I checkpoint) to learn static semantics, composition and text alignment — images are abundant, cheap and high quality, giving the model a “static prior”;

2. Joint image-video training: add short clips (usually 3-10 s, 256-512p) mixed with images (video often only 5-20% of the batch), training the 3D modules (3D conv or causal attention) for motion and frame consistency;

3. HD fine-tuning: raise resolution (256→480→720p→1080p) and duration (4 s→16 s→longer) progressively with curated high-quality data at a small learning rate, layering high-frequency texture onto the learned motion distribution without exploding compute.

Aspect-ratio bucketing: samples are grouped into buckets by aspect ratio/resolution (16:9, 9:16, 1:1…), and every batch is single-bucket — avoiding geometric distortion from naive resizing; standard in SD/Sora pipelines.

Motion filtering: optical flow (e.g. RAFT) or frame differencing estimates each clip's dynamism; static, subtitle-only and low-motion samples are dropped (flow magnitude below a threshold), also helping filter misaligned text-video pairs.

training or continuing text-to-video models; interviews probe per-stage freezing strategies, data mixing ratios and HD fine-tune details.

direct end-to-end training at high resolution/long duration is data-starved and compute-prohibitive; the progressive recipe learns the bulk of the distribution first (static + low-res motion) then fine detail, while bucketing and flow filtering clean the data — the de facto optimal video-diffusion training pipeline.

Test Your Knowledge: Practice Questions for "Progressive Video Training"

Single choice pitfall questions with instant feedback and mistake tracking.