M2-013M2: Classical Machine LearningRegularization (L1 / L2)Medium
⚡Flashcard Drill➔Mastery:
Explain Why Dropout Acts as a Regularizer Through Ensemble and Feature Noise Perspectives
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Dropout acts as an implicit ensemble of sub-networks sharing weights, while simultaneously functioning as data-dependent adaptive feature noise that prevents co-adaptation among neurons.
📌 Key Takeaways
- •1. Ensemble Interpretation: During training, randomly dropping units with probability samples one of thinned subnetworks; at test time, multiplying weights by approximates the geometric ensemble average.
- •2. Anti-Co-adaptation Interpretation: Neurons cannot rely on specific companion neurons to correct errors, forcing every unit to learn independently useful, robust internal representations.
- •3. Connection to L2: In linear models, Dropout is mathematically equivalent to adaptive L2 weight decay.
📐 Mathematical Derivations
Proof of equivalence between Dropout and L2 weight decay (Wager et al., 2013): Consider linear regression . Apply dropout by multiplying feature by Bernoulli mask . The objective is . Decomposing via the Law of Total Variance: . Since , the second term simplifies to: . This is an adaptive L2 regularization where features with higher variance are regularized more aggressively!
🏭 Production Trade-offs
Industrial practices: (1) Inverted Dropout: Multiplies activations by during training: , allowing test-time inference to run unmodified without weight scaling. (2) In modern LLM Transformers, Dropout is often disabled () during large-scale pretraining because massive datasets already provide natural regularization and dropout slows compute throughput.
⚠️ Common Interview Pitfalls
- ✕Forgetting to call
model.eval()at inference time (leaving dropout enabled, injecting stochastic noise into predictions). - ✕Using standard dropout alongside Batch Normalization without careful positioning (variance shift can destabilize training).
🎯 Interviewer Follow-up Inquiries
- ?Why does Inverted Dropout eliminate the need for scaling adjustments during model evaluation?
- ?Why has modern LLM pretraining largely abandoned Dropout in favor of weight decay and data scale?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚