M6-051M6: Multimodal & Generative ModelsDiffusion Models Foundations (DDPM)Hard
⚡Flashcard Drill➔Mastery:
Continuous-Time Diffusion Formulations: Forward/Reverse SDEs and Probability Flow ODEs
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Continuous-time diffusion models data generation through forward and reverse Stochastic Differential Equations (SDEs), associated with a unique deterministic Probability Flow ODE that shares identical marginal probability densities.
📌 Key Takeaways
- •Forward SDE: models continuous noise injection via , progressively transforming data into a tractable Gaussian prior
- •Reverse-time SDE: Anderson's theorem reverses the diffusion trajectory using score function , enabling generative sampling from noise
- •Probability Flow ODE: a completely deterministic ordinary differential equation sharing identical marginal distributions with the reverse SDE, enabling exact likelihood computation and high-speed ODE integration
📐 Mathematical Derivations
Mathematical Mechanism: 1. The Continuous-Time Forward SDE (Song et al., 2021): Let be standard Brownian motion (Wiener process). The forward diffusion is governed by Itô SDE: where is the drift coefficient and is the diffusion coefficient. (a) Variance Preserving (VP) SDE (equivalent to DDPM): (b) Variance Exploding (VE) SDE (equivalent to SMLD): 2. The Generative Reverse-Time SDE: By Anderson's reverse-time theorem, the reverse process running from to is also an Itô SDE: where is a backward Wiener process, and is the score function approximated by neural network . 3. The Probability Flow ODE: Song et al. proved that for every diffusion SDE, there exists an associated deterministic ODE whose trajectories share the exact same marginal probability densities : 4. Exact Likelihood Computation via Instantaneous Change of Variables: Using the Probability Flow ODE, exact log-likelihood is computed by integrating the trace of the Jacobian along the deterministic trajectory: Trace computation is evaluated efficiently using the Hutchinson stochastic trace estimator .
🏭 Production Trade-offs
Deep Dive & Engineering Trade-offs: ① SDE vs ODE Sampling Dynamics: (a) Reverse SDE (Stochastic): Continually injects random Wiener noise during sampling. Stochasticity acts as a self-correcting error barrier; if an intermediate step overshoots, random perturbations push trajectories back toward high-density manifolds. Yields higher visual diversity and photorealism at 50-100 steps. (b) Probability Flow ODE (Deterministic): Completely deterministic mapping between latent space and image space. Enables high-speed solvers (DPM-Solver, UniPC) to generate images in 15-20 steps, and unlocks exact latent inversion and latent interpolation. ② Exact Likelihood Calculation: Unlike standard GANs or discrete VAEs where log-likelihood is approximated or intractable, the Probability Flow ODE allows exact, non-variational evaluation of data log-likelihood via the continuous change of variables theorem. ③ Unification of Generation Paradigms: The SDE/ODE framework unifies DDPM, Score-Based SMLD, Normalizing Flows, and continuous Flow Matching within a single differential geometry foundation. ⑤ Interview Strategy: Write the general forward SDE , state Anderson's reverse SDE formula, derive the Probability Flow ODE equation by halving the diffusion term, contrast stochastic SDE error-correction against deterministic ODE acceleration, and explain exact likelihood computation via the continuous change-of-variables trace formula.
⚠️ Common Interview Pitfalls
- ✕Confusing the backward Wiener process in the reverse SDE with forward Brownian motion; time flows backwards from to
- ✕Attempting to perform exact latent inversion using the stochastic reverse SDE rather than the deterministic Probability Flow ODE
- ✕Assuming the Probability Flow ODE has different marginal probability distributions than the stochastic reverse SDE; marginals are identical
🎯 Interviewer Follow-up Inquiries
- ?Why do the stochastic reverse SDE and the deterministic Probability Flow ODE produce identical marginal probability distributions at all timesteps?
- ?How does the Hutchinson trace estimator enable scalable log-likelihood computation via the instantaneous change-of-variables formula?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚