M3-058M3: Deep Learning FoundationsGradient Vanishing & ExplosionHard
⚡Flashcard Drill➔Mastery:

Gradient Conflict in Multi-Task Learning and Remediation Techniques

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M3-059 →
📐 Mathematical Definition
cos⁡ϕij=gi⋅gj∥gi∥∥gj∥<0 ⇒ conflict;gi←gi−gi⋅gj∥gj∥2gj (PCGrad)\cos\phi_{ij}=\frac{g_i\cdot g_j}{\|g_i\|\|g_j\|}<0\ \Rightarrow\ \text{conflict};\qquad g_i\leftarrow g_i-\frac{g_i\cdot g_j}{\|g_j\|^2}g_j\ (\text{PCGrad})
⚡ One-Sentence Core Answer
Gradients from competing tasks point in opposing directions (⟨gi,gj⟩<0\langle g_i, g_j \rangle < 0), causing destructive cancellation; mitigate via PCGrad orthogonal projection and GradNorm balance.

📌 Key Takeaways

  • •
    Gradient conflict condition: cosine similarity cos⁡(gi,gj)<0\cos(g_i, g_j) < 0; updates for task AA directly degrade task BB
  • •
    PCGrad (Yu et al., 2020): projects conflicting gradients onto the normal plane of the conflicting task (gi←gi−gi⋅gj∥gj∥2gjg_i \leftarrow g_i - \frac{g_i \cdot g_j}{\|g_j\|^2} g_j)
  • •
    GradNorm (Chen et al., 2018): dynamically balances task loss weights wtw_t to equalize learning speeds across tasks

📐 Mathematical Derivations

Mathematical Formulations:
Let total multi-task loss be L=∑t=1TwtLt\mathcal{L} = \sum_{t=1}^T w_t \mathcal{L}_t. The shared parameter gradient is g=∑t=1Twtgtg = \sum_{t=1}^T w_t g_t.
If ⟨gi,gj⟩<0\langle g_i, g_j \rangle < 0, the update Δθ=−ηg\Delta \theta = -\eta g has negative projection along gig_i or gjg_j, meaning progress on one task forces regression on the other (the 'seesaw' effect).
① Projecting Conflicting Gradients (PCGrad):
For a random pair of tasks (i,j)(i, j) with gradients gi,gjg_i, g_j:
If ⟨gi,gj⟩<0\langle g_i, g_j \rangle < 0, replace gig_i with its orthogonal projection onto the normal plane of gjg_j:
giproj=gi−⟨gi,gj⟩∥gj∥22gjg_i^{\text{proj}} = g_i - \frac{\langle g_i, g_j \rangle}{\|g_j\|_2^2} g_j.
Repeat for all task pairs. This ensures ⟨giproj,gj⟩=0\langle g_i^{\text{proj}}, g_j \rangle = 0, eliminating negative interference while preserving shared directional components.
② CAGrad (Conflict-Averse Gradient Descent):
Finds a consensus direction within a ball around the average gradient that maximizes worst-case task improvement.

🏭 Production Trade-offs

Computational overhead: PCGrad requires computing task gradients gtg_t separately (requiring TT backward passes through shared parameters) and performing O(T2)O(T^2) projection inner products, increasing training time by 2×2\times.
⚠️ Common Interview Pitfalls
  • ✕
    Attempting to resolve directional gradient conflicts solely by adjusting static scalar task weights wtw_t
  • ✕
    Applying multi-task gradient projection on single-task objectives where sub-losses are simply components of a single unified goal
🎯 Interviewer Follow-up Inquiries
  • ?
    How does PCGrad ensure order-invariance when projecting across more than two conflicting tasks?
  • ?
    What is the difference between magnitude conflict and directional conflict in multi-task optimization?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM3-057: How to Detect and Isolate Gradient Anomalies: Diagnostic Signals and Workflows📋 Back to BankNext →M3-059: How Residual Connections and Normalization Cooperate to Guarantee Gradient Flow