Double Descent

Double descent (Belkin et al. 2019) describes test error evolving with model capacity/parameter count in two descending segments. In the underparametrized regime (parameters $p$ < samples $n$) error first falls then rises with capacity — the classical U-shape; near the interpolation threshold ($p \approx n$, where the model can just memorize every training point) error peaks, because the model is fitting noise; in the overparametrized regime ($p > n$) test error keeps decreasing with capacity — the second descent. Key mechanism: when overparameterized, infinitely many interpolating (zero training error) solutions exist, and the learning algorithm (e.g. gradient descent) picks the minimum-norm one, whose implicit regularization gives good generalization — benign overfitting.

explains why modern deep learning prefers bigger models — networks with far more parameters than samples still generalize; interviews ask how it relates to the classical bias-variance U-shape and why interpolation + implicit regularization does not violate the no-free-lunch theorem.

the classical U-shape implies models necessarily degrade beyond some capacity, contradicting the empirical success of LLMs and large CNNs; double descent shows generalization is governed jointly by capacity and the solution the algorithm selects: in the overparametrized regime the minimum-norm interpolator acts as implicit regularization, and test error improves monotonically with capacity — consistent with scaling laws (params ↑ → loss ↓), provided data and optimizer cooperate (e.g. SGD favoring flat minima, kernel methods under random features).

Sketch the double-descent curve: mark the underparametrized regime, interpolation threshold and overparametrized regime; contrast with the classical U-shape.

Why is error worst near the interpolation threshold ($p \approx n$)? What is the model doing to training noise there?

Why does generalization improve in the overparametrized regime? What is the link between min-norm interpolation and implicit regularization?

Why don't large models (params ≫ samples) degrade along the U-shape? How does this connect to scaling laws?

Does double descent violate the no-free-lunch theorem? How does it reconcile with the bias-variance decomposition?