Gradient Checking

Approximate the derivative numerically and compare it with the analytical gradient: central difference $f'(x) \approx \frac{f(x+\epsilon) - f(x-\epsilon)}{2\epsilon}$, error $O(\epsilon^2)$ (one-sided differencing is only $O(\epsilon)$), with $\epsilon$ typically $10^{-4}$–$10^{-5}$ (smaller and floating-point noise dominates). Measure with relative error $\frac{|g_{num} - g_{ana}|}{\max(|g_{num}|, |g_{ana}|)}$: use the $10^{-7}$ order for shallow nets (≤3 layers) and the $10^{-3}$ order for deep/large networks, where tiny numerical errors accumulate.

verifying the backward pass when hand-writing backprop, implementing custom autograd Functions, or reproducing a paper's new layer/loss; interviews ask for the formula, the error order, and the thresholds.

backward-pass bugs are subtle — training still “learns” (later layers absorb the error), so the bug is nearly invisible; gradient checking turns implementation correctness into an automated numeric test. Caveats: disable dropout/BN randomness, use only a few samples, check one parameter at a time (two forward passes per parameter), so it is run when implementing new ops, not repeatedly during training.

Write the central-difference gradient formula. Why is the error $O(\epsilon^2)$?

How is relative error computed? Why do shallow and deep networks need different thresholds?

Why disable dropout/BN randomness and use tiny samples during gradient checking?

When is gradient checking appropriate? Why can't it run constantly?

What goes wrong if $\epsilon$ is too large or too small?