M6-084M6: Multimodal & Generative ModelsControllable Generation & Image EditingHard
⚡Flashcard Drill➔Mastery:
Customized Subject Generation: DreamBooth vs Textual Inversion vs LoRA
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Subject-driven personalization injects new visual concepts from 3-5 images into diffusion models, balancing parameter efficiency and prompt steerability across Textual Inversion, DreamBooth, and LoRA.
📌 Key Takeaways
- •Textual Inversion (Gal et al.): freezes all diffusion weights and optimizes a single continuous word embedding token , yielding a tiny 4KB footprint but limited expressivity
- •DreamBooth (Ruiz et al.): fine-tunes all diffusion backbone weights alongside a unique rare identifier token using class-specific prior preservation loss to prevent catastrophic concept forgetting
- •LoRA personalization: parameterizes weight updates as low-rank matrices (), achieving high subject fidelity and prompt steerability with modular 50MB adapter checkpoints
📐 Mathematical Derivations
Mathematical Mechanism: 1. Textual Inversion Formulation: Freezes UNet/DiT and text encoder . Introduces pseudo-word with learnable embedding vector : Operates entirely in word embedding space; cannot modify internal visual feature representations. 2. DreamBooth Formulation: Pairs rare token ('a [V] dog') with Class-Specific Prior Preservation Loss: where is the generic class prompt ('a dog') and are model-generated images of the class. The regularization loss prevents language drift and protects the general prior from collapsing onto the subject. 3. LoRA Personalization (Hu et al.): Decomposes weight deltas: . Freezes base weights and trains only , achieving of DreamBooth's visual fidelity while preventing base model parameter destruction.
🏭 Production Trade-offs
Deep Dive & Engineering Trade-offs: ① The Subject Fidelity vs Prompt Steerability Frontier: (a) Textual Inversion: Low subject fidelity (struggles with intricate patterns), but high prompt steerability (easily placed in novel poses and environments). (b) DreamBooth (Full FT): High subject fidelity, but easily overfits; the subject often copies the exact camera angle, lighting, or background of the 4 training photos, refusing to obey prompts ('a [V] dog running on the moon'). (c) LoRA: The industrial champion; strikes the optimal balance between facial fidelity and prompt flexibility. ② Prior Preservation Loss Rationale: Without class prior preservation , fine-tuning on a specific bulldog causes the model's concept of 'dog' to collapse into that specific bulldog. The model loses the ability to generate other dog breeds. Generating 200 synthetic class images and interleaving them during fine-tuning preserves general distribution variance. ③ Checkpoint Storage and Serving: DreamBooth creates a full checkpoint per subject (prohibitive for consumer cloud services with millions of users). LoRA creates a adapter file that can be dynamically mounted at runtime via multi-LoRA serving. ⑤ Interview Strategy: Formulate the optimization variables across all three methods ( vs full vs ), write DreamBooth's prior preservation loss equation, explain how prior preservation prevents language drift, and detail the fidelity vs steerability trade-off.
⚠️ Common Interview Pitfalls
- ✕Fine-tuning DreamBooth without class-specific prior preservation loss, causing catastrophic language drift on the subject's super-category
- ✕Over-training LoRA on personal photos (> 1500 steps on 5 images), causing the model to memorize background artifacts and refuse prompt instructions
- ✕Using common words (like 'dog' or 'friend') as the unique identifier token instead of rare token sequences (like 'sks' or '[V]')
🎯 Interviewer Follow-up Inquiries
- ?Why is class-specific prior preservation loss mathematically required to prevent category collapse during DreamBooth fine-tuning?
- ?What causes Textual Inversion to underperform LoRA and DreamBooth when capturing fine-grained structural textures and faces?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚