M6-051M6: Multimodal & Generative ModelsDiffusion Models Foundations (DDPM)Hard
⚡Flashcard Drill➔Mastery:

Continuous-Time Diffusion Formulations: Forward/Reverse SDEs and Probability Flow ODEs

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M6-052 →
📐 Mathematical Definition
SDE: dx=[f−12g2∇log⁡p]dt+g2dwˉ;ODE: dx=[f−12g2∇log⁡p]dt\text{SDE}:\ dx=[f-\tfrac12g^2\nabla\log p]dt+\sqrt{g^2}d\bar w;\qquad \text{ODE}:\ dx=[f-\tfrac12g^2\nabla\log p]dt
⚡ One-Sentence Core Answer
Continuous-time diffusion models data generation through forward and reverse Stochastic Differential Equations (SDEs), associated with a unique deterministic Probability Flow ODE that shares identical marginal probability densities.

📌 Key Takeaways

  • •
    Forward SDE: models continuous noise injection via  dx=f(x,t)dt+g(t)dw \,dx = f(x, t)dt + g(t)dw\,, progressively transforming data into a tractable Gaussian prior
  • •
    Reverse-time SDE: Anderson's theorem reverses the diffusion trajectory using score function  ∇xlog⁡pt(x) \,\nabla_x \log p_t(x)\,, enabling generative sampling from noise
  • •
    Probability Flow ODE: a completely deterministic ordinary differential equation sharing identical marginal distributions  pt(x) \,p_t(x)\, with the reverse SDE, enabling exact likelihood computation and high-speed ODE integration

📐 Mathematical Derivations

Mathematical Mechanism: 1. The Continuous-Time Forward SDE (Song et al., 2021): Let ww be standard Brownian motion (Wiener process). The forward diffusion is governed by Itô SDE: dx=f(x,t)dt+g(t)dw,t∈[0,T]dx = f(x, t) dt + g(t) dw, \quad t \in [0, T] where f(x,t)f(x, t) is the drift coefficient and g(t)g(t) is the diffusion coefficient. (a) Variance Preserving (VP) SDE (equivalent to DDPM): f(x,t)=−12β(t)x,g(t)=β(t)f(x, t) = -\frac{1}{2} \beta(t) x, \quad g(t) = \sqrt{\beta(t)} (b) Variance Exploding (VE) SDE (equivalent to SMLD): f(x,t)=0,g(t)=d[σ2(t)]dtf(x, t) = 0, \quad g(t) = \sqrt{\frac{d[\sigma^2(t)]}{dt}} 2. The Generative Reverse-Time SDE: By Anderson's reverse-time theorem, the reverse process running from t=Tt=T to t=0t=0 is also an Itô SDE: dx=[f(x,t)−g(t)2∇xlog⁡pt(x)]dt+g(t)dwˉdx = \left[ f(x, t) - g(t)^2 \nabla_x \log p_t(x) \right] dt + g(t) d\bar{w} where dwˉd\bar{w} is a backward Wiener process, and ∇xlog⁡pt(x)\nabla_x \log p_t(x) is the score function approximated by neural network sθ(x,t)s_\theta(x, t). 3. The Probability Flow ODE: Song et al. proved that for every diffusion SDE, there exists an associated deterministic ODE whose trajectories share the exact same marginal probability densities pt(x)p_t(x): dxdt=f(x,t)−12g(t)2∇xlog⁡pt(x)\frac{dx}{dt} = f(x, t) - \frac{1}{2} g(t)^2 \nabla_x \log p_t(x) 4. Exact Likelihood Computation via Instantaneous Change of Variables: Using the Probability Flow ODE, exact log-likelihood log⁡p0(x0)\log p_0(x_0) is computed by integrating the trace of the Jacobian along the deterministic trajectory: log⁡p0(x0)=log⁡pT(xT)+∫0T∇x⋅(f(x,t)−12g2(t)sθ(x,t))dt\log p_0(x_0) = \log p_T(x_T) + \int_0^T \nabla_x \cdot \left( f(x, t) - \frac{1}{2} g^2(t) s_\theta(x, t) \right) dt Trace computation is evaluated efficiently using the Hutchinson stochastic trace estimator Ev[vTJv]\mathbb{E}_{v}[v^T J v].

🏭 Production Trade-offs

Deep Dive & Engineering Trade-offs: ① SDE vs ODE Sampling Dynamics: (a) Reverse SDE (Stochastic): Continually injects random Wiener noise during sampling. Stochasticity acts as a self-correcting error barrier; if an intermediate step overshoots, random perturbations push trajectories back toward high-density manifolds. Yields higher visual diversity and photorealism at 50-100 steps. (b) Probability Flow ODE (Deterministic): Completely deterministic mapping between latent space N(0,I)\mathcal{N}(0, I) and image space. Enables high-speed solvers (DPM-Solver, UniPC) to generate images in 15-20 steps, and unlocks exact latent inversion and latent interpolation. ② Exact Likelihood Calculation: Unlike standard GANs or discrete VAEs where log-likelihood is approximated or intractable, the Probability Flow ODE allows exact, non-variational evaluation of data log-likelihood via the continuous change of variables theorem. ③ Unification of Generation Paradigms: The SDE/ODE framework unifies DDPM, Score-Based SMLD, Normalizing Flows, and continuous Flow Matching within a single differential geometry foundation. ⑤ Interview Strategy: Write the general forward SDE dx=fdt+gdwdx = f dt + g dw, state Anderson's reverse SDE formula, derive the Probability Flow ODE equation by halving the diffusion term, contrast stochastic SDE error-correction against deterministic ODE acceleration, and explain exact likelihood computation via the continuous change-of-variables trace formula.
⚠️ Common Interview Pitfalls
  • ✕
    Confusing the backward Wiener process dwˉd\bar{w} in the reverse SDE with forward Brownian motion; time flows backwards from TT to 00
  • ✕
    Attempting to perform exact latent inversion using the stochastic reverse SDE rather than the deterministic Probability Flow ODE
  • ✕
    Assuming the Probability Flow ODE has different marginal probability distributions pt(x)p_t(x) than the stochastic reverse SDE; marginals are identical
🎯 Interviewer Follow-up Inquiries
  • ?
    Why do the stochastic reverse SDE and the deterministic Probability Flow ODE produce identical marginal probability distributions pt(x)p_t(x) at all timesteps?
  • ?
    How does the Hutchinson trace estimator enable scalable log-likelihood computation via the instantaneous change-of-variables formula?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM6-050: Conditioning Mechanisms in Diffusion Models: Cross-Attention, Channel Concatenation, and AdaLN📋 Back to BankNext →M6-052: Classifier-Free Guidance (CFG): Mathematical Formulation and Conditional Steering
中文速览与面试要点

解释扩散模型的 SDE 与 ODE 采样。

反向 SDE(随机,DDPM/Euler-a)与概率流 ODE(确定,DDIM/DPM-Solver)有相同边缘分布;ODE 可大步长、SDE 多样性更高。

核心要点

  • 数学机理: 统一框架(Song 等 2021) ——前向过程写成 SDE:dx=f(x,t)dt+g(t)dW(把数据变成噪声); 反向 SDE (从噪声生成数据):dx=[f(x,t)−g²(t)∇_x log p_t(x)]dt+g(t)d w,其中 ∇log p_t 由 score 网络提供(与 ε 成正比)。 概率流 ODE ——对同一个 SDE,存在一个 确定性 ODE :dx=[f(x,t)−½g²(t)∇log p_t(x)]dt; 关键性质 ——该 ODE 与反向 SDE 有 相同的边缘分布 p_t(x)(即在每个时刻 t,两者的样本分布相同)。 因此 ——(a) 用 SDE 采样(随机)或 ODE 采样(确定)都能得到正确的最终分布;(b) 差异在'路径'与'多样性':SDE 的路径有随机性(不同轨迹)、ODE 的路径确定(给定起点唯一轨迹)。 DDPM 与 DDIM 的对应 ——(a) DDPM 对应某个特定 SDE 的离散化(Euler-Maruyama),每步加噪;(b) DDIM 对应概率流 ODE 的一阶离散化(σ=0);(c) 故'DDPM 与 DDIM 是同一框架的两种采样'。 实践中的 SDE 变体 ——(a) Euler-a (Karras 等)——在 SDE 中引入'时间步相关的噪声尺度'(churn),使各步的噪声注入可控;(b) Langevin 修正 ——在某些步用 Langevin 动力学提升质量。 选择依据 ——(a) 确定性(ODE) ——可复现、可用大步长(少步采样)、支持 inversion(图像编辑)、支持插值; 缺点 ——多样性较低(需靠不同起点)。(b) 随机(SDE) ——多样性更高(每步加噪引入变化);在'高分辨率生成'时有时质量更好(有研究显示 SDE 在细节上更优); 缺点 ——不可复现、步数受限(随机项要求小步长)。 混合方案 ——(a) 部分步用 SDE、部分用 ODE ;(b) Euler-a 的 churn 调度 (在中间步注入噪声);(c) DPM-Solver++ (ODE 求解器 + 可选的 SDE 修正)。 度量 ——(a) FID/质量 ;(b) 多样性 (如 recall、LPIPS 分布);(c) 步数与延迟 。 实践建议 ——(a) 快速采样 → ODE(DPM-Solver,10~20 步);(b) 编辑/复现 → ODE;(c) 追求细节/多样性 → SDE(Euler-a);(d) 按任务选。
  • 深度剖析与工程权衡:① '同边缘分布'是统一框架的核心 ——它解释了'为什么两种采样都正确';面试中能指出这一点是深度理解的标志。② 'ODE 可大步长' ——因为确定性 ODE 的数值积分可用高阶方法(误差可控);而 SDE 的随机项要求小步长。③ 'SDE 多样性更高' ——随机项引入变化;但在'同一 prompt 生成多样结果'上,ODE 也可通过不同起点实现(故差异不绝对)。④ 'inversion 需要 ODE' ——DDIM inversion 依赖确定性映射(可逆);这是'图像编辑'的基础。⑤ 'Euler-a 的 churn' ——它是在 SDE 框架下'可控地注入噪声'(在中间步加噪以提升细节);这是'随机 vs 确定'的连续调节旋钮。⑥ 面试要点 ——被问'SDE vs ODE 采样',应给出' 反向 SDE(随机,DDPM 系)vs 概率流 ODE(确定,DDIM/DPM-Solver) '与' 同边缘分布、ODE 可大步长、SDE 多样性更高 ';能指出'ODE 是 inversion/编辑的基础'是深度理解的标志。
  • 反向 SDE:含随机项(每步加噪),对应 DDPM 系
  • 概率流 ODE:去掉随机项,对应 DDIM/DPM-Solver
  • 两者边缘分布相同;ODE 可用大步长、SDE 多样性更高

常见陷阱

  • 以为 ODE 与 SDE 采样会得到不同分布(同边缘分布)
  • 用随机采样做精确 inversion