M3-007M3: Deep Learning FoundationsBackprop & AutodiffMedium
⚡Flashcard Drill➔Mastery:
Gradient Checking: Principles, Finite Differences, and Implementation Details
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Verify analytical autograd implementations by comparing computed gradients against two-sided finite difference numerical approximations using relative error metrics.
📌 Key Takeaways
- •Two-sided difference: with truncation error
- •Relative error threshold: evaluate ; score indicates correctness
- •Precision prerequisite: must run in float64 (double precision) to avoid catastrophic numerical cancellation
📐 Mathematical Derivations
Mathematical Foundations:
① Taylor Expansion Truncation Error:
- One-sided difference: .
- Two-sided symmetric difference: , and .
Subtracting yields: . Error drops quadratically with perturbation (typically in float64).
② Relative Error Metric:
.
- : Excellent agreement.
- to : Caution, verify if non-differentiable kinks (ReLU, max-pooling) are active.
- : Definite gradient implementation bug.
① Taylor Expansion Truncation Error:
- One-sided difference: .
- Two-sided symmetric difference: , and .
Subtracting yields: . Error drops quadratically with perturbation (typically in float64).
② Relative Error Metric:
.
- : Excellent agreement.
- to : Caution, verify if non-differentiable kinks (ReLU, max-pooling) are active.
- : Definite gradient implementation bug.
🏭 Production Trade-offs
Implementation details: Never run gradient checks in float32; roundoff errors in single precision dominate differences, causing false alarms. In PyTorch, use
torch.autograd.gradcheck(func, inputs, eps=1e-6, atol=1e-4).⚠️ Common Interview Pitfalls
- ✕Running gradient checking in float32, confusing floating-point subtraction cancellation with implementation bugs
- ✕Performing gradient check at points of non-differentiability (e.g., exactly at for ReLU), where left and right derivatives diverge
🎯 Interviewer Follow-up Inquiries
- ?Why is the two-sided finite difference approximation strictly superior to the one-sided difference ?
- ?How do you perform gradient checking on layers with stochastic operations like Dropout?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚