M6-056M6: Multimodal & Generative ModelsGuidance & Fast Sampling (CFG / DDIM)Hard
⚡Flashcard Drill➔Mastery:
Stochasticity vs Reproducibility in Diffusion Sampling: Seeds, Noise Invariance, and Hardware Drift
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Deterministic diffusion ODE samplers guarantee mathematical reproducibility given identical noise seeds, while stochastic SDE samplers inject continuous random perturbations that produce varied outputs.
📌 Key Takeaways
- •Deterministic sampling (DDIM, DPM-Solver): ODE flow establishes a deterministic 1-to-1 bijection between initial noise and generated image
- •Stochastic sampling (DDPM, Euler-a): continuously injects Wiener noise perturbations at every step, creating high sample diversity but non-reproducible trajectories
- •Hardware and kernel non-determinism: identical seeds can still produce diverging images across different GPU architectures, CUDA versions, or non-deterministic atomic operations
📐 Mathematical Derivations
Mathematical Mechanism: 1. Deterministic Mapping: In a Probability Flow ODE solver: where is the numerical integration operator. Because the ODE vector field is Lipschitz continuous, the trajectory is uniquely determined by initial value and model parameters . If the random seed generating is fixed, is mathematically invariant: 2. Stochastic SDE Trajectory: In ancestral or reverse SDE sampling: The total trajectory entropy is: Even if initial seed is locked, intermediate perturbations generate a distribution of outputs. 3. Sources of Hardware Non-Determinism: Even with fixed seeds, floating-point non-associativity causes divergence: (a) Atomic Add Operations: CUDA atomic additions in FlashAttention or cuBLAS accumulate tensor values in non-deterministic thread order. (b) Tensor Core Precision: TF32 or FP16 rounding differences between Ampere (A100) and Hopper (H100) architectures create minor numerical perturbations at that amplify exponentially along non-linear diffusion trajectories.
🏭 Production Trade-offs
Deep Dive & Engineering Trade-offs: ① The Butterfly Effect in Diffusion Trajectories: Because the reverse trajectory unrolls through 30-50 non-linear deep network layers, a tiny numerical discrepancy of at step 1 expands across steps: . By step 0, two runs on different GPU types with the exact same seed may generate images with different hair configurations or background elements. Achieving true bitwise reproducibility requires setting
torch.use_deterministic_algorithms(True) and locking identical GPU hardware. ② Seed Invariance in Creative Workflows: Deterministic ODE solvers are essential for user-facing creative tools: users expect that locking the seed allows them to tweak single words in the prompt ('change blue jacket to red jacket') without altering the subject's face or composition. ③ Batch Size Invariance Trap: Generating 4 images in a single batch (batch_size=4) versus generating 4 images sequentially with sequential seeds (batch_size=1) can produce different images if the random number generator (RNG) strides differently across batch tensor dimensions. Using individual generator instances per sample (torch.Generator(device='cuda').manual_seed(...)) guarantees batch-size independence. ⑤ Interview Strategy: Contrast deterministic ODE mapping against stochastic SDE entropy accumulation, explain floating-point atomic add non-determinism, describe the chaotic amplification of minor numerical shifts, and present per-sample generator seeding.⚠️ Common Interview Pitfalls
- ✕Assuming identical random seeds will produce bitwise identical images across different GPU architectures (e.g., RTX 3090 vs A100)
- ✕Using stochastic samplers (Euler-a) for image editing workflows that require deterministic latent seed consistency
- ✕Failing to instantiate independent per-sample PyTorch
torch.Generatorinstances, causing batch-size-dependent seed drift
🎯 Interviewer Follow-up Inquiries
- ?Why does a minor floating-point discrepancy () at early sampling steps amplify into major visual divergence in generated images?
- ?How does instantiating dedicated per-sample PyTorch CPU/CUDA generators guarantee batch-size independent reproducibility?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚