M3-005M3: Deep Learning FoundationsBackprop & AutodiffHard
⚡Flashcard Drill➔Mastery:
Computational Cost of Higher-Order Derivatives and When They Are Needed
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Materializing the full Hessian scales quadratically in memory and cubically in compute; efficient frameworks use Hessian-Vector Products (HVP) to compute directional curvature in backward passes.
📌 Key Takeaways
- •Full Hessian cost: ; impossible to store or invert for deep networks ()
- •Hessian-Vector Product (HVP): Pearlmutter's trick computes using exactly one extra backward pass
- •Use cases: MAML (meta-learning), WGAN-GP (gradient penalty), Natural Gradient Descent, second-order optimizers (K-FAC)
📐 Mathematical Derivations
Mathematical Formulations:
① The Full Hessian Barrier: For loss , . For a 1B parameter model, storing requires bytes Exabytes of VRAM.
② Pearlmutter's Trick (Hessian-Vector Product): Compute the directional derivative of gradient along vector : .
In PyTorch autograd:
```python
grad = torch.autograd.grad(loss, params, create_graph=True)[0]
grad_v = torch.sum(grad * v)
hvp = torch.autograd.grad(grad_v, params)[0] # Exact H * v without forming H
Cost: Exactly backward passes ( time and memory), completely bypassing the matrix bottleneck.
① The Full Hessian Barrier: For loss , . For a 1B parameter model, storing requires bytes Exabytes of VRAM.
② Pearlmutter's Trick (Hessian-Vector Product): Compute the directional derivative of gradient along vector : .
In PyTorch autograd:
```python
grad = torch.autograd.grad(loss, params, create_graph=True)[0]
grad_v = torch.sum(grad * v)
hvp = torch.autograd.grad(grad_v, params)[0] # Exact H * v without forming H
`Cost: Exactly backward passes ( time and memory), completely bypassing the matrix bottleneck.
🏭 Production Trade-offs
Industrial applications: ① WGAN-GP: Enforces 1-Lipschitz continuity via gradient penalty , requiring backprop through gradients. ② Model-Agnostic Meta-Learning (MAML): Differentiates through an inner gradient descent update step to optimize meta-loss .
⚠️ Common Interview Pitfalls
- ✕Setting
create_graph=Falsewhen computing gradients that will be differentiated again in a loss penalty, resulting in detached gradients - ✕Attempting to compute explicit Hessians using
torch.autograd.functional.hessianon deep networks, immediately causing OOM crashes
🎯 Interviewer Follow-up Inquiries
- ?How does Pearlmutter's operator compute Hessian-vector products with zero numerical approximation error?
- ?Why does First-Order MAML (FOMAML) omit second derivatives, and how much performance is sacrificed?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚