M3-047M3: Deep Learning FoundationsRegularization & Training TricksEasy
⚡Flashcard Drill➔Mastery:

Major Regularization Techniques in Deep Learning

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M3-048 →
📐 Mathematical Definition
Lreg=L(θ)+λΩ(θ);ΩL2=12∥θ∥2\mathcal{L}_{\text{reg}}=\mathcal{L}(\theta)+\lambda\Omega(\theta);\qquad \Omega_{\text{L2}}=\tfrac12\|\theta\|^2
⚡ One-Sentence Core Answer
Regularization introduces structural priors to constrain capacity: parameter penalties (weight decay), architectural dropouts, data augmentations (Mixup), and targets (label smoothing).

📌 Key Takeaways

  • •
    Parameter regularization: L2 weight decay (Gaussian prior), L1 (Laplace sparsity), spectral norm
  • •
    Stochastic perturbations: Dropout, DropPath (stochastic depth), noisy activations
  • •
    Data & Target regularization: CutMix, Mixup, Label Smoothing, Virtual Adversarial Training

📐 Mathematical Derivations

Taxonomy of Deep Regularization:
① Parameter Regularization: Constrains weight norm. Lreg=L0+λ2∥W∥22\mathcal{L}_{\text{reg}} = \mathcal{L}_0 + \frac{\lambda}{2} \|W\|_2^2. Corresponds to zero-mean Gaussian prior in MAP estimation.
② Stochastic Architectural Perturbations:
- Dropout: Multiplies activations by Bernoulli mask m∼Bernoulli(1−p)m \sim \text{Bernoulli}(1-p), preventing feature co-adaptation.
- DropPath / Stochastic Depth: Bypasses entire residual blocks.
③ Data Augmentation & Interpolation:
- Mixup: Enforces linear interpolation between inputs and labels: x~=λxi+(1−λ)xj,  y~=λyi+(1−λ)yj\tilde{x} = \lambda x_i + (1-\lambda)x_j, \; \tilde{y} = \lambda y_i + (1-\lambda)y_j.
- CutMix: Replaces spatial patches with image cutouts.
④ Label Regularization:
- Label Smoothing: Softens hard 0/1 one-hot labels to yksmooth=(1−ϵ)yk+ϵKy_k^{\text{smooth}} = (1 - \epsilon) y_k + \frac{\epsilon}{K}, preventing softmax overconfidence and infinite logit growth.
⑤ Optimization Dynamics: Early stopping, Stochastic Weight Averaging (SWA), and Exponential Moving Average (EMA).

🏭 Production Trade-offs

Modern paradigm shift: In small datasets, heavy regularization (Dropout, Weight Decay) is mandatory to prevent overfitting. In massive LLM pretraining (>15> 15T tokens), explicit regularization like Dropout is typically set to 0 because underfitting data is the primary challenge, and Dropout wastes model capacity.
⚠️ Common Interview Pitfalls
  • ✕
    Using aggressive Dropout (p=0.5p=0.5) in foundation LLM pretraining, significantly degrading training efficiency and token throughput
  • ✕
    Tuning multiple regularizers simultaneously without controlled ablation studies
🎯 Interviewer Follow-up Inquiries
  • ?
    Why do modern LLMs (like LLaMA and GPT-3) set Dropout to 0 during pretraining?
  • ?
    How does Label Smoothing mathematically prevent weights in the output projection layer from growing to infinity?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM3-046: The One-Cycle Policy and Cyclical Learning Rates: Mechanisms and Super-Convergence📋 Back to BankNext →M3-048: Why Weight Decay is Decoupled from Learning Rate