M3-053M3: Deep Learning FoundationsRegularization & Training TricksHard
⚡Flashcard Drill➔Mastery:
Dropout Usage in Transformers and LLMs: Differences Across Modalities and Scales
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Dropout prevents co-adaptation in small-data regimes, but is universally set to 0 in large language model pre-training to preserve model capacity and maximize token throughput.
📌 Key Takeaways
- •Inverted Dropout: where ; leaves inference untouched
- •Standard Transformers (BERT/T5): uses in attention probabilities, hidden projections, and embeddings
- •Modern Foundation LLMs (LLaMA/GPT-3): sets dropout to ; massive web corpora make overfitting nonexistent during single-epoch pretraining
📐 Mathematical Derivations
Formulation and System Impact:
① Inverted Dropout (Srivastava et al., 2014):
- Training: , where . The scaling factor ensures , eliminating the need to scale activations during inference.
- Ensemble Interpretation: Acts as an implicit geometric ensemble of sub-networks sharing weights.
② Why Foundation LLMs Abandon Dropout ($p=0$):
- Capacity Under-utilization: Pretraining models on 15+ trillion tokens is strictly under-fitting regime (single pass over data). Adding dropout artificially reduces effective model parameter capacity by .
- Compute & Bandwidth Overhead: Generating random PRNG masks and performing element-wise masking increases GPU memory traffic and prevents operator fusion in fused attention/MLP kernels.
- Fine-tuning: In low-resource supervised fine-tuning (SFT) or LoRA with small sample counts (), dropout is reintroduced () to prevent memorization.
① Inverted Dropout (Srivastava et al., 2014):
- Training: , where . The scaling factor ensures , eliminating the need to scale activations during inference.
- Ensemble Interpretation: Acts as an implicit geometric ensemble of sub-networks sharing weights.
② Why Foundation LLMs Abandon Dropout ($p=0$):
- Capacity Under-utilization: Pretraining models on 15+ trillion tokens is strictly under-fitting regime (single pass over data). Adding dropout artificially reduces effective model parameter capacity by .
- Compute & Bandwidth Overhead: Generating random PRNG masks and performing element-wise masking increases GPU memory traffic and prevents operator fusion in fused attention/MLP kernels.
- Fine-tuning: In low-resource supervised fine-tuning (SFT) or LoRA with small sample counts (), dropout is reintroduced () to prevent memorization.
🏭 Production Trade-offs
Sub-layer placement: When dropout is used in Transformers, it is applied at: (1) token embeddings, (2) attention weights after softmax, and (3) residual output projections. It is never applied inside LayerNorm/RMSNorm blocks.
⚠️ Common Interview Pitfalls
- ✕Leaving dropout enabled () during large-scale foundation pretraining, wasting millions of dollars in compute capacity
- ✕Forgetting to call
model.eval(), which leaves random dropout active during inference evaluation
🎯 Interviewer Follow-up Inquiries
- ?Why is Inverted Dropout preferred over standard original dropout in modern deep learning libraries?
- ?Under what specific fine-tuning conditions should dropout be reintroduced in LLMs?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚