Dropout

Dropout randomly zeroes neurons with probability $p$ during training to prevent co-adaptation. The standard implementation is Inverted Dropout: sample a Bernoulli mask $m \sim \text{Bernoulli}(1-p)$ and scale $y = \frac{1}{1-p} m \odot x$ so that $\mathbb{E}[y] = \mathbb{E}[x]$; at inference no scaling is applied and the full network is used.

the main regularizer for fully-connected and attention layers (attention output and FFN in Transformers), typically $p = 0.1$–$0.3$ for LLM pretraining/finetuning; interviews ask about the train/inference formula difference, why inference must disable it, and its order with BN.

it performs implicit ensembling over $2^n$ subnetworks, substantially reducing overfitting (original paper: test error reduced by roughly 4–5%). Inverted dropout moves the scaling into training so inference is cost-free — this is PyTorch's default (torch.nn.Dropout acts only in train mode). When combined with BN, apply dropout BEFORE normalization: BN's re-normalization would absorb dropout's scaling and noise, weakening the regularization.

Write the inverted-dropout training formula $y = \frac{1}{1-p} m \odot x$ ($m \sim \text{Bernoulli}(1-p)$) and explain why inference needs no scaling to preserve the expectation.

What is the mathematical relationship between naive dropout (scaling at test) and inverted dropout (scaling at train)? Why does every framework implement the inverted version?

Why does dropout regularize: its link to ensemble learning and co-adaptation — and which layers is it usually applied to?

What happens if dropout is not disabled at inference (model.eval())? What problems arise from train/inference inconsistency?

What order should be used when combining dropout with BatchNorm, and why is dropout usually applied before BN?