← Back to Multimodal Mind Map
👁️ MultimodalID: video-training-pipeline

Progressive Video Training

视频渐进训练
🎯Core Definition
Text-to-video training follows a three-stage progressive recipe (Sora, Stable Video Diffusion, Kling and others), gradually introducing the video distribution:
📌Overview
1. Image pretraining: train on massive image-text pairs (or load a T2I checkpoint) to learn static semantics, composition and text alignment — images are abundant, cheap and high quality, giving the model a “static prior”;
📌Overview
2. Joint image-video training: add short clips (usually 3-10 s, 256-512p) mixed with images (video often only 5-20% of the batch), training the 3D modules (3D conv or causal attention) for motion and frame consistency;
📌Overview
3. HD fine-tuning: raise resolution (256→480→720p→1080p) and duration (4 s→16 s→longer) progressively with curated high-quality data at a small learning rate, layering high-frequency texture onto the learned motion distribution without exploding compute.
📌Overview
Aspect-ratio bucketing: samples are grouped into buckets by aspect ratio/resolution (16:9, 9:16, 1:1…), and every batch is single-bucket — avoiding geometric distortion from naive resizing; standard in SD/Sora pipelines.
📌Overview
Motion filtering: optical flow (e.g. RAFT) or frame differencing estimates each clip's dynamism; static, subtitle-only and low-motion samples are dropped (flow magnitude below a threshold), also helping filter misaligned text-video pairs.
💡Use Cases
training or continuing text-to-video models; interviews probe per-stage freezing strategies, data mixing ratios and HD fine-tune details.
⚡Key Problems Solved
direct end-to-end training at high resolution/long duration is data-starved and compute-prohibitive; the progressive recipe learns the bulk of the distribution first (static + low-res motion) then fine detail, while bucketing and flow filtering clean the data — the de facto optimal video-diffusion training pipeline.
🎯Core Interview Exam Points
Fundamentals · Production Scenarios · Deep Dive
Core Def
Explain the core definition, underlying mechanism/math, and key principles of Progressive Video Training.
Use Cases
In production engineering, what are the primary use cases for Progressive Video Training and what core pain points does it solve?
🔥 High-Frequency Deep-Dive Questions
1
What does each of the three training stages learn, and why is the order fixed?
2
What does aspect-ratio bucketing solve, and how is a single batch organized?
3
Why filter training clips with optical flow, and how is the threshold set?
4
Learning-rate and data-selection choices in the HD fine-tune stage?
5
Why does image pretraining benefit video generation?
📖 In-depth Guide:📄 diffusion-models →
Updated 2026-08-12
🎯
Verify Mastery: Practice Exam Questions for "Progressive Video Training"
Take pitfall multiple-choice questions with derivation options; mistakes are automatically logged.
🚀 Start Card Practice ➔
← Previous CardDiffusion Training Objectives (ε/v-pred, Zero-SNR)Next Card →Audio Representation

🔗 More Multimodal Knowledge Cards

引用出处: TalentMe 450+ 工业级 AI 架构考点知识库 (Tech Vault) · 遵循 CC-BY-4.0 知识共享协议
URL: https://talentme.airsota.com/resources/tech/roadmap/mindmap/Multimodal/card/video-training-pipeline
中文速览与面试要点

视频渐进训练 (Progressive Video Training)

定义: 文生视频采用三阶段渐进式训练(Sora、Stable Video Diffusion、Kling 等通用配方),逐步引入视频分布: 1. 图像预训练: 先用海量图文对训练(或加载现成文生图权重),学会静态语义、构图与文本对齐——图像数据多、质量高、成本低,为模型提供“静态先验”; 2. 图文视频联合训练: 加入短视频片段(通常 3-10 秒、256-512p),与图像按比例混合(视频占比通常仅 5%-20%),训练 3D 模块(3D 卷积或因果注意力)学习运动与帧间一致性; 3. 高清微调: 逐步提高分辨率(256→480→720p→1080p)与时长(4s→16s→更长),用高质量精选数据、小学习率微调,在已学好的运动分布上“叠加”高频纹理细节,避免高分辨率直接训练的算力爆炸。 宽高比分桶(aspect-ratio bucketing): 训练前按宽高比/分辨率将样本分桶(如 16:9、9:16、1:1),一个 batch 内所有样本同桶同尺寸,避免粗暴 resize 造成几何畸变与构图偏差,是 SD/Sora 训练标配。 光流过滤(motion filtering): 用光流模型(如 RAFT)或帧差估计每段视频的动态程度,丢弃静态、纯字幕、低运动样本(光流幅值低于阈值),并辅助筛选文本-视频对齐差的样本,保证训练数据“动起来”且与文本相关。 使用场景: 文生视频模型训练/续训;面试常追问三阶段的冻结策略、数据配比与高清微调细节。 解决的问题: 直接高分辨率、长时长端到端训练视频模型,数据稀缺且算力爆炸;渐进式配方让模型先学分布主体(静态+低分辨率运动)、再学边缘细节,分桶与光流过滤净化数据,以最小算力达到最好的时序质量与构图,是视频扩散训练工程的通用最优解。

核心要点

  • 视频训练三阶段各自训什么?为什么顺序不能反?
  • 宽高比分桶解决什么问题?一个 batch 内的样本如何组织?
  • 为什么要用光流过滤训练数据?阈值如何设定?
  • 高清微调阶段如何设置学习率与数据选择?
  • 图像预训练为什么对视频生成有帮助?