M3-051M3: Deep Learning FoundationsRegularization & Training TricksHard
⚡Flashcard Drill➔Mastery:
Stochastic Weight Averaging (SWA) and Weight Ensembling (Model Soups)
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
SWA averages checkpoint weights along a cyclical/constant learning rate trajectory to discover flatter minima; Model Soups blend weights of multiple fine-tuned models with zero extra inference cost.
📌 Key Takeaways
- •SWA mechanism: sampled during late-stage training at high/cyclical LR
- •Geometry of SWA: points on the boundary of a wide loss valley average into the flat interior, boosting test generalization
- •Model Soups: linearly interpolates fine-tuned weights sharing the same pre-trained initialization without ensemble inference latency
📐 Mathematical Derivations
Mathematical Formulation (Izmailov et al., UAI 2018; Wortsman et al., ICML 2022):
① Stochastic Weight Averaging (SWA):
Standard SGD with learning rate decay converges to a point near the periphery of a flat loss basin. SWA maintains a cyclical or constant high learning rate schedule in late epochs, causing parameter trajectories to explore the basin boundaries. Checkpoint weights are collected every epochs:
.
Because loss surfaces are locally convex, the geometric mean of boundary points lies deep inside the flat interior, achieving substantially lower test error and high robustness to covariate shift.
② Model Soups:
When fine-tuning the same pre-trained checkpoint with diverse hyperparameters (learning rate, augmentation, seeds), models remain in the same linear basin. Blending weights directly (where ) matches or outperforms traditional multi-model output ensembling while incurring zero additional inference memory or latency overhead.
① Stochastic Weight Averaging (SWA):
Standard SGD with learning rate decay converges to a point near the periphery of a flat loss basin. SWA maintains a cyclical or constant high learning rate schedule in late epochs, causing parameter trajectories to explore the basin boundaries. Checkpoint weights are collected every epochs:
.
Because loss surfaces are locally convex, the geometric mean of boundary points lies deep inside the flat interior, achieving substantially lower test error and high robustness to covariate shift.
② Model Soups:
When fine-tuning the same pre-trained checkpoint with diverse hyperparameters (learning rate, augmentation, seeds), models remain in the same linear basin. Blending weights directly (where ) matches or outperforms traditional multi-model output ensembling while incurring zero additional inference memory or latency overhead.
🏭 Production Trade-offs
Post-processing requirement: After computing , the running mean and variance of all BatchNorm layers must be updated by passing training data through the model in a forward pass (
torch.optim.swa_utils.update_bn). LayerNorm models do not require this calibration step.⚠️ Common Interview Pitfalls
- ✕Averaging weights of models trained from completely different random initializations; they occupy distinct loss basins, causing complete prediction collapse
- ✕Evaluating SWA on BatchNorm architectures without updating running statistics, producing terrible benchmark scores
🎯 Interviewer Follow-up Inquiries
- ?Why does linear weight interpolation fail between two models trained from different random initializations?
- ?How does Greedy Model Soups decide which fine-tuned checkpoints to include in the average?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚