M6-051M6: Multimodal & Generative ModelsDiffusion Models Foundations (DDPM)Hard
⚡Flashcard Drill➔Mastery:

Continuous-Time Diffusion Formulations: Forward/Reverse SDEs and Probability Flow ODEs

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M6-052 →
📐 Mathematical Definition
SDE: dx=[f−12g2∇log⁡p]dt+g2dwˉ;ODE: dx=[f−12g2∇log⁡p]dt\text{SDE}:\ dx=[f-\tfrac12g^2\nabla\log p]dt+\sqrt{g^2}d\bar w;\qquad \text{ODE}:\ dx=[f-\tfrac12g^2\nabla\log p]dt
⚡ One-Sentence Core Answer
Continuous-time diffusion models data generation through forward and reverse Stochastic Differential Equations (SDEs), associated with a unique deterministic Probability Flow ODE that shares identical marginal probability densities.

📌 Key Takeaways

  • •
    Forward SDE: models continuous noise injection via  dx=f(x,t)dt+g(t)dw \,dx = f(x, t)dt + g(t)dw\,, progressively transforming data into a tractable Gaussian prior
  • •
    Reverse-time SDE: Anderson's theorem reverses the diffusion trajectory using score function  ∇xlog⁡pt(x) \,\nabla_x \log p_t(x)\,, enabling generative sampling from noise
  • •
    Probability Flow ODE: a completely deterministic ordinary differential equation sharing identical marginal distributions  pt(x) \,p_t(x)\, with the reverse SDE, enabling exact likelihood computation and high-speed ODE integration

📐 Mathematical Derivations

Mathematical Mechanism: 1. The Continuous-Time Forward SDE (Song et al., 2021): Let ww be standard Brownian motion (Wiener process). The forward diffusion is governed by Itô SDE: dx=f(x,t)dt+g(t)dw,t∈[0,T]dx = f(x, t) dt + g(t) dw, \quad t \in [0, T] where f(x,t)f(x, t) is the drift coefficient and g(t)g(t) is the diffusion coefficient. (a) Variance Preserving (VP) SDE (equivalent to DDPM): f(x,t)=−12β(t)x,g(t)=β(t)f(x, t) = -\frac{1}{2} \beta(t) x, \quad g(t) = \sqrt{\beta(t)} (b) Variance Exploding (VE) SDE (equivalent to SMLD): f(x,t)=0,g(t)=d[σ2(t)]dtf(x, t) = 0, \quad g(t) = \sqrt{\frac{d[\sigma^2(t)]}{dt}} 2. The Generative Reverse-Time SDE: By Anderson's reverse-time theorem, the reverse process running from t=Tt=T to t=0t=0 is also an Itô SDE: dx=[f(x,t)−g(t)2∇xlog⁡pt(x)]dt+g(t)dwˉdx = \left[ f(x, t) - g(t)^2 \nabla_x \log p_t(x) \right] dt + g(t) d\bar{w} where dwˉd\bar{w} is a backward Wiener process, and ∇xlog⁡pt(x)\nabla_x \log p_t(x) is the score function approximated by neural network sθ(x,t)s_\theta(x, t). 3. The Probability Flow ODE: Song et al. proved that for every diffusion SDE, there exists an associated deterministic ODE whose trajectories share the exact same marginal probability densities pt(x)p_t(x): dxdt=f(x,t)−12g(t)2∇xlog⁡pt(x)\frac{dx}{dt} = f(x, t) - \frac{1}{2} g(t)^2 \nabla_x \log p_t(x) 4. Exact Likelihood Computation via Instantaneous Change of Variables: Using the Probability Flow ODE, exact log-likelihood log⁡p0(x0)\log p_0(x_0) is computed by integrating the trace of the Jacobian along the deterministic trajectory: log⁡p0(x0)=log⁡pT(xT)+∫0T∇x⋅(f(x,t)−12g2(t)sθ(x,t))dt\log p_0(x_0) = \log p_T(x_T) + \int_0^T \nabla_x \cdot \left( f(x, t) - \frac{1}{2} g^2(t) s_\theta(x, t) \right) dt Trace computation is evaluated efficiently using the Hutchinson stochastic trace estimator Ev[vTJv]\mathbb{E}_{v}[v^T J v].

🏭 Production Trade-offs

Deep Dive & Engineering Trade-offs: ① SDE vs ODE Sampling Dynamics: (a) Reverse SDE (Stochastic): Continually injects random Wiener noise during sampling. Stochasticity acts as a self-correcting error barrier; if an intermediate step overshoots, random perturbations push trajectories back toward high-density manifolds. Yields higher visual diversity and photorealism at 50-100 steps. (b) Probability Flow ODE (Deterministic): Completely deterministic mapping between latent space N(0,I)\mathcal{N}(0, I) and image space. Enables high-speed solvers (DPM-Solver, UniPC) to generate images in 15-20 steps, and unlocks exact latent inversion and latent interpolation. ② Exact Likelihood Calculation: Unlike standard GANs or discrete VAEs where log-likelihood is approximated or intractable, the Probability Flow ODE allows exact, non-variational evaluation of data log-likelihood via the continuous change of variables theorem. ③ Unification of Generation Paradigms: The SDE/ODE framework unifies DDPM, Score-Based SMLD, Normalizing Flows, and continuous Flow Matching within a single differential geometry foundation. ⑤ Interview Strategy: Write the general forward SDE dx=fdt+gdwdx = f dt + g dw, state Anderson's reverse SDE formula, derive the Probability Flow ODE equation by halving the diffusion term, contrast stochastic SDE error-correction against deterministic ODE acceleration, and explain exact likelihood computation via the continuous change-of-variables trace formula.
⚠️ Common Interview Pitfalls
  • ✕
    Confusing the backward Wiener process dwˉd\bar{w} in the reverse SDE with forward Brownian motion; time flows backwards from TT to 00
  • ✕
    Attempting to perform exact latent inversion using the stochastic reverse SDE rather than the deterministic Probability Flow ODE
  • ✕
    Assuming the Probability Flow ODE has different marginal probability distributions pt(x)p_t(x) than the stochastic reverse SDE; marginals are identical
🎯 Interviewer Follow-up Inquiries
  • ?
    Why do the stochastic reverse SDE and the deterministic Probability Flow ODE produce identical marginal probability distributions pt(x)p_t(x) at all timesteps?
  • ?
    How does the Hutchinson trace estimator enable scalable log-likelihood computation via the instantaneous change-of-variables formula?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM6-050: Conditioning Mechanisms in Diffusion Models: Cross-Attention, Channel Concatenation, and AdaLN📋 Back to BankNext →M6-052: Classifier-Free Guidance (CFG): Mathematical Formulation and Conditional Steering