M2-099M2: Classical Machine LearningRegularization (L1 / L2)Hard
⚡Flashcard Drill➔Mastery:
Stochastic Depth (DropPath) vs Standard Dropout: Mechanisms and Differences
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Dropout randomly zeroes out individual activations, whereas Stochastic Depth drops entire residual blocks during training, shortening effective depth and perfectly preserving BatchNorm statistics.
📌 Key Takeaways
- •Granularity: Dropout drops individual neurons/channels; Stochastic Depth drops complete residual branches
- •Path shortening: Stochastic Depth allows gradients to flow directly through skip connections:
- •BatchNorm compatibility: dropping entire branches preserves intra-layer activation distributions, avoiding variance shifts
📐 Mathematical Derivations
Mathematical formulation: In a residual block, let the transformation be .
Stochastic Depth introduces a Bernoulli random variable with survival probability : .
- Training Mode: When , the sub-network bypasses layer entirely via the identity connection . In inverted form: .
- Linear Decay Schedule: Survival probability typically decreases linearly with depth: , keeping shallow feature extractors intact while heavily regularizing redundant deep layers.
- Difference from Dropout: Dropout scales active neurons by , altering feature covariance and creating internal covariate shifts that conflict with Batch Normalization's running variance estimates. Stochastic Depth executes complete layer bypasses, maintaining stable feature distributions.
Stochastic Depth introduces a Bernoulli random variable with survival probability : .
- Training Mode: When , the sub-network bypasses layer entirely via the identity connection . In inverted form: .
- Linear Decay Schedule: Survival probability typically decreases linearly with depth: , keeping shallow feature extractors intact while heavily regularizing redundant deep layers.
- Difference from Dropout: Dropout scales active neurons by , altering feature covariance and creating internal covariate shifts that conflict with Batch Normalization's running variance estimates. Stochastic Depth executes complete layer bypasses, maintaining stable feature distributions.
🏭 Production Trade-offs
Architecture matching: Standard Dropout is best suited for dense linear heads. Stochastic Depth (often called
DropPath in Vision Transformers) is essential for training ultra-deep ResNets, ConvNeXt, and Vision Transformers (ViT, Swin).⚠️ Common Interview Pitfalls
- ✕Applying DropPath to the identity skip connection rather than the residual transformation branch
- ✕Dropping early layers with high probability, which destroys basic low-level spatial and semantic feature extraction
🎯 Interviewer Follow-up Inquiries
- ?Why does Dropout cause test-time performance degradation when paired immediately before Batch Normalization?
- ?How does Stochastic Depth function as an implicit ensemble of variable-depth networks?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚