M6-052M6: Multimodal & Generative ModelsGuidance & Fast Sampling (CFG / DDIM)Easy
⚡Flashcard Drill➔Mastery:

Classifier-Free Guidance (CFG): Mathematical Formulation and Conditional Steering

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M6-053 →
📐 Mathematical Definition
ϵ^=ϵθ(xt,∅)+s(ϵθ(xt,c)−ϵθ(xt,∅))\hat\epsilon=\epsilon_\theta(x_t,\varnothing)+s\left(\epsilon_\theta(x_t,c)-\epsilon_\theta(x_t,\varnothing)\right)
⚡ One-Sentence Core Answer
Classifier-Free Guidance steers diffusion generation toward prompt conditions by linearly extrapolating between conditional and unconditional score predictions, eliminating external classifier networks.

📌 Key Takeaways

  • •
    Extrapolation formula: combines unconditional prediction  ϵθ(xt,∅) \,\epsilon_\theta(x_t, \emptyset)\, and conditional prediction  ϵθ(xt,c) \,\epsilon_\theta(x_t, c)\, via guidance scale ss:  ϵ^=ϵθ(xt,∅)+s⋅(ϵθ(xt,c)−ϵθ(xt,∅)) \,\hat{\epsilon} = \epsilon_\theta(x_t, \emptyset) + s \cdot (\epsilon_\theta(x_t, c) - \epsilon_\theta(x_t, \emptyset))\,
  • •
    Joint training protocol: randomly drops the conditioning prompt cc with probability puncond∈[0.1,0.2]p_{\text{uncond}} \in [0.1, 0.2] during training, replacing it with empty token ∅\emptyset
  • •
    Score sharpening effect: setting s>1s > 1 amplifies conditional modes and suppresses unconditional background density, boosting text alignment and visual clarity at the expense of sample diversity

📐 Mathematical Derivations

Mathematical Mechanism: 1. Implicit Classifier Bayes Decomposition: By Bayes' rule, the conditional score decomposes into unconditional score plus classifier gradient: ∇xlog⁡p(x∣c)=∇xlog⁡p(x)+∇xlog⁡p(c∣x)\nabla_x \log p(x \mid c) = \nabla_x \log p(x) + \nabla_x \log p(c \mid x) Ho & Salimans (2021) observed that the implicit classifier gradient can be expressed purely using conditional and unconditional generative models: ∇xlog⁡p(c∣x)=∇xlog⁡p(x∣c)−∇xlog⁡p(x)\nabla_x \log p(c \mid x) = \nabla_x \log p(x \mid c) - \nabla_x \log p(x) 2. Guided Score Formulation: Substituting this implicit classifier gradient into guided score with temperature/scale multiplier ss: ∇~xlog⁡ps(x∣c)=∇xlog⁡p(x)+s⋅∇xlog⁡p(c∣x)\tilde{\nabla}_x \log p_s(x \mid c) = \nabla_x \log p(x) + s \cdot \nabla_x \log p(c \mid x) =∇xlog⁡p(x)+s[∇xlog⁡p(x∣c)−∇xlog⁡p(x)]=(1−s)∇xlog⁡p(x)+s∇xlog⁡p(x∣c)= \nabla_x \log p(x) + s \big[ \nabla_x \log p(x \mid c) - \nabla_x \log p(x) \big] = (1 - s) \nabla_x \log p(x) + s \nabla_x \log p(x \mid c) 3. Noise Predictor Parameterization: Since score relates to predicted noise via ∇xtlog⁡p(xt)=−ϵθ1−αˉt\nabla_{x_t} \log p(x_t) = - \frac{\epsilon_\theta}{\sqrt{1-\bar{\alpha}_t}}, the guided noise prediction is: ϵ^θ(xt,c)=ϵθ(xt,∅)+s⋅(ϵθ(xt,c)−ϵθ(xt,∅))=(1−s)ϵθ(xt,∅)+sϵθ(xt,c)\hat{\epsilon}_\theta(x_t, c) = \epsilon_\theta(x_t, \emptyset) + s \cdot \big( \epsilon_\theta(x_t, c) - \epsilon_\theta(x_t, \emptyset) \big) = (1 - s) \epsilon_\theta(x_t, \emptyset) + s \epsilon_\theta(x_t, c) When s=1s = 1, generation reduces to standard conditional diffusion without guidance. When s>1s > 1, the vector difference (ϵ(c)−ϵ(∅))(\epsilon(c) - \epsilon(\emptyset)) pushes the generation trajectory away from generic unconditional data toward the conditioned prompt manifold.

🏭 Production Trade-offs

Deep Dive & Engineering Trade-offs: ① The Dual Forward Pass Tax: Standard CFG requires evaluating the neural network twice per sampling step: once for ϵθ(xt,c)\epsilon_\theta(x_t, c) and once for ϵθ(xt,∅)\epsilon_\theta(x_t, \emptyset). In production serving, the batch size is doubled (2B2B), doubling latency and FLOPs per request. ② The Conditioning Dropout Rate ($p_{\text{uncond}}$): Setting puncondp_{\text{uncond}} too low (<5%< 5\%) prevents the model from learning a robust unconditional prior ϵ(∅)\epsilon(\emptyset), causing CFG extrapolation to produce erratic artifacts. Setting puncondp_{\text{uncond}} too high (>30%> 30\%) degrades conditional text adherence. Industry standard consensus fixes puncond=0.10–0.15p_{\text{uncond}} = 0.10\text{--}0.15. ③ Negative Prompting as Dynamic Unconditional Steering: In standard CFG, the unconditional branch uses an empty string ∅\emptyset. In creative text-to-image tools (Automatic1111, Midjourney), users provide a 'negative prompt' cnegc_{\text{neg}} ('blurry, distorted, extra limbs'). Replacing ϵ(∅)\epsilon(\emptyset) with ϵ(cneg)\epsilon(c_{\text{neg}}) actively repels generated samples from the negative semantic manifold. ④ Interview Strategy: Formulate the Bayesian score decomposition, derive the linear combination ϵ^=ϵ(∅)+s(ϵ(c)−ϵ(∅))\hat{\epsilon} = \epsilon(\emptyset) + s(\epsilon(c) - \epsilon(\emptyset)), explain how training with condition dropout (p=0.1p=0.1) enables single-model dual capability, and explain the negative prompting mechanism.
⚠️ Common Interview Pitfalls
  • ✕
    Setting guidance scale s=1.0s=1.0 and expecting strong text prompt adherence; without guidance (s>1s > 1), complex prompt composition collapses
  • ✕
    Forgetting to train with unconditional condition dropout (puncondp_{\text{uncond}}), making CFG extrapolation impossible at inference time
  • ✕
    Assuming CFG adds no serving compute; CFG doubles the effective forward-pass batch size at every sampling step
🎯 Interviewer Follow-up Inquiries
  • ?
    How does substituting a negative text prompt in place of the empty condition ∅\emptyset in CFG mathematically repel unwanted visual features?
  • ?
    Why is conditioning dropout (puncond≈0.1p_{\text{uncond}} \approx 0.1) during training mandatory for enabling Classifier-Free Guidance at inference time?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM6-051: Continuous-Time Diffusion Formulations: Forward/Reverse SDEs and Probability Flow ODEs📋 Back to BankNext →M6-053: Classifier Guidance vs Classifier-Free Guidance: Gradients, Joint Training, and Trade-offs