M3-058M3: Deep Learning FoundationsGradient Vanishing & ExplosionHard
⚡Flashcard Drill➔Mastery:
Gradient Conflict in Multi-Task Learning and Remediation Techniques
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Gradients from competing tasks point in opposing directions (), causing destructive cancellation; mitigate via PCGrad orthogonal projection and GradNorm balance.
📌 Key Takeaways
- •Gradient conflict condition: cosine similarity ; updates for task directly degrade task
- •PCGrad (Yu et al., 2020): projects conflicting gradients onto the normal plane of the conflicting task ()
- •GradNorm (Chen et al., 2018): dynamically balances task loss weights to equalize learning speeds across tasks
📐 Mathematical Derivations
Mathematical Formulations:
Let total multi-task loss be . The shared parameter gradient is .
If , the update has negative projection along or , meaning progress on one task forces regression on the other (the 'seesaw' effect).
① Projecting Conflicting Gradients (PCGrad):
For a random pair of tasks with gradients :
If , replace with its orthogonal projection onto the normal plane of :
.
Repeat for all task pairs. This ensures , eliminating negative interference while preserving shared directional components.
② CAGrad (Conflict-Averse Gradient Descent):
Finds a consensus direction within a ball around the average gradient that maximizes worst-case task improvement.
Let total multi-task loss be . The shared parameter gradient is .
If , the update has negative projection along or , meaning progress on one task forces regression on the other (the 'seesaw' effect).
① Projecting Conflicting Gradients (PCGrad):
For a random pair of tasks with gradients :
If , replace with its orthogonal projection onto the normal plane of :
.
Repeat for all task pairs. This ensures , eliminating negative interference while preserving shared directional components.
② CAGrad (Conflict-Averse Gradient Descent):
Finds a consensus direction within a ball around the average gradient that maximizes worst-case task improvement.
🏭 Production Trade-offs
Computational overhead: PCGrad requires computing task gradients separately (requiring backward passes through shared parameters) and performing projection inner products, increasing training time by .
⚠️ Common Interview Pitfalls
- ✕Attempting to resolve directional gradient conflicts solely by adjusting static scalar task weights
- ✕Applying multi-task gradient projection on single-task objectives where sub-losses are simply components of a single unified goal
🎯 Interviewer Follow-up Inquiries
- ?How does PCGrad ensure order-invariance when projecting across more than two conflicting tasks?
- ?What is the difference between magnitude conflict and directional conflict in multi-task optimization?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚