Training Debugging

A systematic pipeline for fixing broken training. Step 1 — overfit a single batch: train on 1 (or a few) samples; loss should quickly drop to ~0, otherwise there is an implementation bug — inspect the data pipeline (label misalignment/normalization/shuffling), forward pass (wrong loss/dimensions), backward pass (missing or detached gradients), and the optimizer (parameters actually updating); only after gradients are verified, scale up data and model. Step 2 — loss-curve diagnosis: high flat training loss → underfitting (capacity/lr/features); training loss drops but validation stalls or rises → overfitting (more data/regularization/less capacity); both stalled → lr too small or data issues (label noise/shuffling); oscillation/divergence → lr too high or exploding gradients. Step 3 — gradient/weight statistics: NaN or exploding gradient norms → clipping or lower lr; all-zero weights → initialization or dead ReLUs.

the first response when training diverges, loss is NaN, or validation never improves; interviews ask how you would debug “loss won't drop” or “loss = NaN”.

it turns debugging from black-box tinkering into a layered hunt: the single-batch overfit test answers “is the code correct?” in minutes, then capacity/data/lr issues are diagnosed separately — so you never waste compute tuning hyperparameters on top of a broken implementation.

Steps and rationale of the single-batch overfit test? Which bugs does it isolate first?

Order of checks when training loss does not drop (data pipeline→forward→backward→optimizer)?

Typical loss-curve shapes for underfitting, overfitting, and wrong learning rates?

Common causes of NaN loss and how to debug them?

How do you fix high-train/low-val versus low-train/high-val situations?