Double Descent

Double descent (Belkin et al. 2019) describes test error evolving with model capacity/parameter count in two descending segments. In the underparametrized regime (parameters $p$ < samples $n$) error first falls then rises with capacity — the classical U-shape; near the interpolation threshold ($p \approx n$, where the model can just memorize every training point) error peaks, because the model is fitting noise; in the overparametrized regime ($p > n$) test error keeps decreasing with capacity — the second descent. Key mechanism: when overparameterized, infinitely many interpolating (zero training error) solutions exist, and the learning algorithm (e.g. gradient descent) picks the minimum-norm one, whose implicit regularization gives good generalization — benign overfitting.

explains why modern deep learning prefers bigger models — networks with far more parameters than samples still generalize; interviews ask how it relates to the classical bias-variance U-shape and why interpolation + implicit regularization does not violate the no-free-lunch theorem.

the classical U-shape implies models necessarily degrade beyond some capacity, contradicting the empirical success of LLMs and large CNNs; double descent shows generalization is governed jointly by capacity and the solution the algorithm selects: in the overparametrized regime the minimum-norm interpolator acts as implicit regularization, and test error improves monotonically with capacity — consistent with scaling laws (params ↑ → loss ↓), provided data and optimizer cooperate (e.g. SGD favoring flat minima, kernel methods under random features).

Test Your Knowledge: Practice Questions for "Double Descent"

Single choice pitfall questions with instant feedback and mistake tracking.