M3-007M3: Deep Learning FoundationsBackprop & AutodiffMedium
⚡Flashcard Drill➔Mastery:

Gradient Checking: Principles, Finite Differences, and Implementation Details

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M3-008 →
📐 Mathematical Definition
∂L∂θi≈L(θ+ϵei)−L(θ−ϵei)2ϵ\frac{\partial L}{\partial\theta_i}\approx\frac{L(\theta+\epsilon e_i)-L(\theta-\epsilon e_i)}{2\epsilon}
⚡ One-Sentence Core Answer
Verify analytical autograd implementations by comparing computed gradients against two-sided finite difference numerical approximations using relative error metrics.

📌 Key Takeaways

  • •
    Two-sided difference: f′(x)≈f(x+ϵ)−f(x−ϵ)2ϵf'(x) \approx \frac{f(x + \epsilon) - f(x - \epsilon)}{2\epsilon} with truncation error O(ϵ2)O(\epsilon^2)
  • •
    Relative error threshold: evaluate ∥ganalytical−gnumerical∥2∥ganalytical∥2+∥gnumerical∥2\frac{\|g_{\text{analytical}} - g_{\text{numerical}}\|_2}{\|g_{\text{analytical}}\|_2 + \|g_{\text{numerical}}\|_2}; score <10−7< 10^{-7} indicates correctness
  • •
    Precision prerequisite: must run in float64 (double precision) to avoid catastrophic numerical cancellation

📐 Mathematical Derivations

Mathematical Foundations:
① Taylor Expansion Truncation Error:
- One-sided difference: f(x+ϵ)−f(x)ϵ=f′(x)+O(ϵ)\frac{f(x+\epsilon) - f(x)}{\epsilon} = f'(x) + O(\epsilon).
- Two-sided symmetric difference: f(x+ϵ)=f(x)+f′(x)ϵ+12f′′(x)ϵ2+16f′′′(x)ϵ3+O(ϵ4)f(x+\epsilon) = f(x) + f'(x)\epsilon + \frac{1}{2}f''(x)\epsilon^2 + \frac{1}{6}f'''(x)\epsilon^3 + O(\epsilon^4), and f(x−ϵ)=f(x)−f′(x)ϵ+12f′′(x)ϵ2−16f′′′(x)ϵ3+O(ϵ4)f(x-\epsilon) = f(x) - f'(x)\epsilon + \frac{1}{2}f''(x)\epsilon^2 - \frac{1}{6}f'''(x)\epsilon^3 + O(\epsilon^4).
Subtracting yields: f(x+ϵ)−f(x−ϵ)2ϵ=f′(x)+O(ϵ2)\frac{f(x+\epsilon) - f(x-\epsilon)}{2\epsilon} = f'(x) + O(\epsilon^2). Error drops quadratically with perturbation ϵ\epsilon (typically ϵ∼10−5\epsilon \sim 10^{-5} in float64).
② Relative Error Metric:
Relative Error=∥ganal−gnum∥2max⁡(∥ganal∥2+∥gnum∥2,10−8)\text{Relative Error} = \frac{\|g_{\text{anal}} - g_{\text{num}}\|_2}{\max(\|g_{\text{anal}}\|_2 + \|g_{\text{num}}\|_2, 10^{-8})}.
- <10−7< 10^{-7}: Excellent agreement.
- 10−410^{-4} to 10−710^{-7}: Caution, verify if non-differentiable kinks (ReLU, max-pooling) are active.
- >10−4> 10^{-4}: Definite gradient implementation bug.

🏭 Production Trade-offs

Implementation details: Never run gradient checks in float32; roundoff errors in single precision dominate differences, causing false alarms. In PyTorch, use torch.autograd.gradcheck(func, inputs, eps=1e-6, atol=1e-4).
⚠️ Common Interview Pitfalls
  • ✕
    Running gradient checking in float32, confusing floating-point subtraction cancellation with implementation bugs
  • ✕
    Performing gradient check at points of non-differentiability (e.g., exactly at x=0x=0 for ReLU), where left and right derivatives diverge
🎯 Interviewer Follow-up Inquiries
  • ?
    Why is the two-sided finite difference approximation O(ϵ2)O(\epsilon^2) strictly superior to the one-sided difference O(ϵ)O(\epsilon)?
  • ?
    How do you perform gradient checking on layers with stochastic operations like Dropout?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM3-006: What is Differentiable Programming and How Does It Reshape System Design?📋 Back to BankNext →M3-008: Why All-Zero Weight Initialization Fails in Neural Networks