M3-053M3: Deep Learning FoundationsRegularization & Training TricksHard
⚡Flashcard Drill➔Mastery:

Dropout Usage in Transformers and LLMs: Differences Across Modalities and Scales

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M3-054 →
📐 Mathematical Definition
dropout: h′=h⊙m/(1−p), m∼Bernoulli(1−p)\text{dropout}:\ h'=h\odot m/(1-p),\ m\sim\mathrm{Bernoulli}(1-p)
⚡ One-Sentence Core Answer
Dropout prevents co-adaptation in small-data regimes, but is universally set to 0 in large language model pre-training to preserve model capacity and maximize token throughput.

📌 Key Takeaways

  • •
    Inverted Dropout: y=11−pm⊙xy = \frac{1}{1-p} m \odot x where m∼Bernoulli(1−p)m \sim \text{Bernoulli}(1-p); leaves inference untouched
  • •
    Standard Transformers (BERT/T5): uses p=0.1p = 0.1 in attention probabilities, hidden projections, and embeddings
  • •
    Modern Foundation LLMs (LLaMA/GPT-3): sets dropout to 0.00.0; massive web corpora make overfitting nonexistent during single-epoch pretraining

📐 Mathematical Derivations

Formulation and System Impact:
① Inverted Dropout (Srivastava et al., 2014):
- Training: y=11−p(m⊙x)y = \frac{1}{1 - p} (m \odot x), where mi∼Bernoulli(1−p)m_i \sim \text{Bernoulli}(1 - p). The scaling factor 11−p\frac{1}{1-p} ensures E[y]=x\mathbb{E}[y] = x, eliminating the need to scale activations during inference.
- Ensemble Interpretation: Acts as an implicit geometric ensemble of 2d2^d sub-networks sharing weights.
② Why Foundation LLMs Abandon Dropout ($p=0$):
- Capacity Under-utilization: Pretraining models on 15+ trillion tokens is strictly under-fitting regime (single pass over data). Adding dropout artificially reduces effective model parameter capacity by p%p\%.
- Compute & Bandwidth Overhead: Generating random PRNG masks and performing element-wise masking increases GPU memory traffic and prevents operator fusion in fused attention/MLP kernels.
- Fine-tuning: In low-resource supervised fine-tuning (SFT) or LoRA with small sample counts (N<5000N < 5000), dropout is reintroduced (p=0.05−0.1p=0.05-0.1) to prevent memorization.

🏭 Production Trade-offs

Sub-layer placement: When dropout is used in Transformers, it is applied at: (1) token embeddings, (2) attention weights after softmax, and (3) residual output projections. It is never applied inside LayerNorm/RMSNorm blocks.
⚠️ Common Interview Pitfalls
  • ✕
    Leaving dropout enabled (p=0.1p=0.1) during large-scale foundation pretraining, wasting millions of dollars in compute capacity
  • ✕
    Forgetting to call model.eval(), which leaves random dropout active during inference evaluation
🎯 Interviewer Follow-up Inquiries
  • ?
    Why is Inverted Dropout preferred over standard original dropout in modern deep learning libraries?
  • ?
    Under what specific fine-tuning conditions should dropout be reintroduced in LLMs?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM3-052: Label Smoothing: Role, Mathematical Mechanism, and Hidden Costs📋 Back to BankNext →M3-054: Vanishing and Exploding Gradients: Root Causes, Dynamics, and Modern Mitigations