B5Easy★ Core Essential · Industrial BedrockPart B · Normalization Family

Inverted Dropout with Train/Eval Scaling

⚡Flashcard Drill➔

Scale kept activations by 1/(1-p) during training so expectations match without extra work at inference.

⏱️ Time: 训练期 O(N) 掩码采样与乘法;推理期 O(1) 零计算恒等
💾 Space: 训练期需临时保存 mask 用于反传;推理期 0 额外显存
🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
import numpy as np
def inverted_dropout(x: np.ndarray, p: float = 0.5, training: bool = True) -> np.ndarray:
"""
参数:
x: 输入张量
p: 丢弃概率 (0 <= p < 1),注意是丢弃概率而非保留概率
training: 是否处于训练模式
"""
if not training or p == 0.0:
# 推理阶段直接返回恒等映射,零运算开销
return x
keep_prob = 1.0 - p
# 生成保留掩码 (按 keep_prob 为 1,否则为 0)
mask = (np.random.rand(*x.shape) < keep_prob).astype(x.dtype)
# 核心:除以 keep_prob 进行倒置期望补偿
return
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: Why is Dropout commonly set to 0 during pretraining of modern LLMs such as LLaMA or GPT-3?
Next: Sinusoidal Positional Encoding →
💡

Core Mental Anchor / Mnemonic

Scale by 1/(1-p) during training so eval mode is an exact zero-cost identity pass.

📐 Mathematical Derivation & Core Formula

Train: y=x⊙m1−p,m∼Bernoulli(1−p);Eval: y=x\text{Train: } y = \frac{x \odot m}{1 - p}, \quad m \sim \mathrm{Bernoulli}(1 - p); \quad \text{Eval: } y = x
### Inverted Scaling Derivation
Standard Dropout zeroes activations with probability pp during training: y=x⊙my = x \odot m where E[m]=1−p\mathbb{E}[m] = 1 - p, giving E[y]=(1−p)x\mathbb{E}[y] = (1 - p)x. Test time then has to multiply activations by (1−p)(1 - p) to match scale.
Modern solution (Inverted Dropout):
Perform the scaling during training by dividing by (1−p)(1 - p):
E[ytrain]=E[x⊙m1−p]=x⋅(1−p)1−p=x\mathbb{E}[y_{\text{train}}] = \mathbb{E}\left[\frac{x \odot m}{1 - p}\right] = \frac{x \cdot (1 - p)}{1 - p} = x
Because expected scale is already preserved during training, inference simplifies to $y = x$ with zero extra compute or scaling ops.

🔄 Tensor Dimensions & Shape Flow

Train: (B, S, D) -> Bernoulli mask (0/1) -> x * mask / (1-p) -> (B, S, D); Eval: identity pass-through

🛡️ Numerical Stability & Pitfalls Checklist

  • Keep drop probability p strictly < 1.0 to avoid division by zero.
  • Skip random masking and scaling completely during evaluation.
  • Propagate backward gradients through the same training mask / keep_prob.

💻 Reference Code Implementation

import numpy as np

def inverted_dropout(x: np.ndarray, p: float = 0.5, training: bool = True) -> np.ndarray:
    """
    参数:
        x: 输入张量
        p: 丢弃概率 (0 <= p < 1),注意是丢弃概率而非保留概率
        training: 是否处于训练模式
    """
    if not training or p == 0.0:
        # 推理阶段直接返回恒等映射,零运算开销
        return x
    
    keep_prob = 1.0 - p
    # 生成保留掩码 (按 keep_prob 为 1,否则为 0)
    mask = (np.random.rand(*x.shape) < keep_prob).astype(x.dtype)
    
    # 核心:除以 keep_prob 进行倒置期望补偿
    return (x * mask) / keep_prob

🧪 Runnable Assertions & Validation

Copy and run directly in Python / Jupyter:

import numpy as np
x = np.ones((1000, 1000))
out_train = inverted_dropout(x, p=0.3, training=True)
# 训练期均值数学期望应严格保持为 1.0
assert np.isclose(out_train.mean(), 1.0, atol=1e-2)
# 检查置零比例约为 30%
assert np.isclose((out_train == 0).mean(), 0.3, atol=1e-2)
out_eval = inverted_dropout(x, p=0.3, training=False)
assert np.array_equal(out_eval, x), "推理阶段必须为完全恒等"
print("✓ Inverted Dropout 自测通过")

🎯 Core Follow-up Inquiries & Reference Answers

Q1:Why is Dropout commonly set to 0 during pretraining of modern LLMs such as LLaMA or GPT-3?
Web-scale pretraining corpora span trillions of tokens with minimal epoch repetition, so overfitting risk is low. Dropping activations also adds memory-bandwidth overhead for storing random masks and can slightly degrade training capacity efficiency.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
← Prev: Group Normalization & Instance NormalizationBack to Part B & All 105Next: Sinusoidal Positional Encoding →