🧠 Deep LearningID: dropout

Dropout

Q: Write the inverted-dropout training formula $y = \frac{1}{1-p} m \odot x$ ($m \sim \text{Bernoulli}(1-p)$) and explain why inference needs no scaling to preserve the expectation.

【Dropout 核心考点 1】Write the inverted-dropout training formula $y = \frac{1}{1-p} m \odot x$ ($m \sim \text{Bernoulli}(1-p)$) and explain why inference needs no scaling to preserve the expectation.。技术解析：Definition: Dropout randomly zeroes neurons with probability p during training to prevent co-adaptation. The standard implementation is Inverted Dropout: sample a Bernoulli mask m \sim \text{Bernoulli}(1-p) and scale y = \frac{1}{1-p} m \odot x so that \mathbb{E}[y] = \mathbb{E}[... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/DL/card/dropout

Q: What is the mathematical relationship between naive dropout (scaling at test) and inverted dropout (scaling at train)? Why does every framework implement the inverted version?

【Dropout 核心考点 2】What is the mathematical relationship between naive dropout (scaling at test) and inverted dropout (scaling at train)? Why does every framework implement the inverted version?。技术解析：Definition: Dropout randomly zeroes neurons with probability p during training to prevent co-adaptation. The standard implementation is Inverted Dropout: sample a Bernoulli mask m \sim \text{Bernoulli}(1-p) and scale y = \frac{1}{1-p} m \odot x so that \mathbb{E}[y] = \mathbb{E}[... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/DL/card/dropout

Q: Why does dropout regularize: its link to ensemble learning and co-adaptation — and which layers is it usually applied to?

【Dropout 核心考点 3】Why does dropout regularize: its link to ensemble learning and co-adaptation — and which layers is it usually applied to?。技术解析：Definition: Dropout randomly zeroes neurons with probability p during training to prevent co-adaptation. The standard implementation is Inverted Dropout: sample a Bernoulli mask m \sim \text{Bernoulli}(1-p) and scale y = \frac{1}{1-p} m \odot x so that \mathbb{E}[y] = \mathbb{E}[... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/DL/card/dropout

Q: What happens if dropout is not disabled at inference (model.eval())? What problems arise from train/inference inconsistency?

【Dropout 核心考点 4】What happens if dropout is not disabled at inference (model.eval())? What problems arise from train/inference inconsistency?。技术解析：Definition: Dropout randomly zeroes neurons with probability p during training to prevent co-adaptation. The standard implementation is Inverted Dropout: sample a Bernoulli mask m \sim \text{Bernoulli}(1-p) and scale y = \frac{1}{1-p} m \odot x so that \mathbb{E}[y] = \mathbb{E}[... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/DL/card/dropout

Q: What order should be used when combining dropout with BatchNorm, and why is dropout usually applied before BN?

【Dropout 核心考点 5】What order should be used when combining dropout with BatchNorm, and why is dropout usually applied before BN?。技术解析：Definition: Dropout randomly zeroes neurons with probability p during training to prevent co-adaptation. The standard implementation is Inverted Dropout: sample a Bernoulli mask m \sim \text{Bernoulli}(1-p) and scale y = \frac{1}{1-p} m \odot x so that \mathbb{E}[y] = \mathbb{E}[... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/DL/card/dropout

🎯Core Definition

Dropout randomly zeroes neurons with probability

p

during training to prevent co-adaptation. The standard implementation is Inverted Dropout: sample a Bernoulli mask

m \sim \text{Bernoulli}(1-p)

and scale

y = \frac{1}{1-p} m \odot x

so that

\mathbb{E}[y] = \mathbb{E}[x]

; at inference no scaling is applied and the full network is used.

💡Use Cases

the main regularizer for fully-connected and attention layers (attention output and FFN in Transformers), typically

p = 0.1

–

0.3

for LLM pretraining/finetuning; interviews ask about the train/inference formula difference, why inference must disable it, and its order with BN.

⚡Key Problems Solved

it performs implicit ensembling over

2^n

subnetworks, substantially reducing overfitting (original paper: test error reduced by roughly 4–5%). Inverted dropout moves the scaling into training so inference is cost-free — this is PyTorch's default (torch.nn.Dropout acts only in train mode). When combined with BN, apply dropout BEFORE normalization: BN's re-normalization would absorb dropout's scaling and noise, weakening the regularization.

🎯5 High-Frequency Exam Points

Write the inverted-dropout training formula

y = \frac{1}{1-p} m \odot x

(

m \sim \text{Bernoulli}(1-p)

) and explain why inference needs no scaling to preserve the expectation.

What is the mathematical relationship between naive dropout (scaling at test) and inverted dropout (scaling at train)? Why does every framework implement the inverted version?

Why does dropout regularize: its link to ensemble learning and co-adaptation — and which layers is it usually applied to?

What happens if dropout is not disabled at inference (model.eval())? What problems arise from train/inference inconsistency?

What order should be used when combining dropout with BatchNorm, and why is dropout usually applied before BN?

📖 In-depth Guide:📄 normalization-and-regularization →

Updated 2026-08-12

🎯

Test Your Knowledge: Practice Questions for "Dropout"

Single choice pitfall questions with instant feedback and mistake tracking.

🚀 Start Card Practice ➔

← Previous CardNorm Methods Comparison Next Card →Label Smoothing

🔗 More Deep Learning Knowledge Cards

Activation Functions Adam & AdamW Autograd Compute Graph Batch Normalization