M3-051M3: Deep Learning FoundationsRegularization & Training TricksHard
⚡Flashcard Drill➔Mastery:

Stochastic Weight Averaging (SWA) and Weight Ensembling (Model Soups)

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M3-052 →
📐 Mathematical Definition
θSWA=1k∑i=1kθi;y^=1k∑i=1kfi(x) (ensemble)\theta_{\text{SWA}}=\frac1k\sum_{i=1}^{k}\theta_i;\qquad \hat y=\frac1k\sum_{i=1}^{k}f_i(x)\ (\text{ensemble})
⚡ One-Sentence Core Answer
SWA averages checkpoint weights along a cyclical/constant learning rate trajectory to discover flatter minima; Model Soups blend weights of multiple fine-tuned models with zero extra inference cost.

📌 Key Takeaways

  • •
    SWA mechanism: θSWA=1K∑k=1Kθk\theta_{\text{SWA}} = \frac{1}{K} \sum_{k=1}^K \theta_k sampled during late-stage training at high/cyclical LR
  • •
    Geometry of SWA: points on the boundary of a wide loss valley average into the flat interior, boosting test generalization
  • •
    Model Soups: linearly interpolates fine-tuned weights sharing the same pre-trained initialization without ensemble inference latency

📐 Mathematical Derivations

Mathematical Formulation (Izmailov et al., UAI 2018; Wortsman et al., ICML 2022):
① Stochastic Weight Averaging (SWA):
Standard SGD with learning rate decay converges to a point near the periphery of a flat loss basin. SWA maintains a cyclical or constant high learning rate schedule η\eta in late epochs, causing parameter trajectories to explore the basin boundaries. Checkpoint weights are collected every cc epochs:
θSWA(k)=kθSWA(k−1)+θkk+1\theta_{\text{SWA}}^{(k)} = \frac{k \theta_{\text{SWA}}^{(k-1)} + \theta_k}{k + 1}.
Because loss surfaces are locally convex, the geometric mean of boundary points lies deep inside the flat interior, achieving substantially lower test error and high robustness to covariate shift.
② Model Soups:
When fine-tuning the same pre-trained checkpoint with diverse hyperparameters (learning rate, augmentation, seeds), models remain in the same linear basin. Blending weights directly θsoup=∑i=1Mwiθi\theta_{\text{soup}} = \sum_{i=1}^M w_i \theta_i (where ∑wi=1\sum w_i = 1) matches or outperforms traditional multi-model output ensembling while incurring zero additional inference memory or latency overhead.

🏭 Production Trade-offs

Post-processing requirement: After computing θSWA\theta_{\text{SWA}}, the running mean and variance of all BatchNorm layers must be updated by passing training data through the model in a forward pass (torch.optim.swa_utils.update_bn). LayerNorm models do not require this calibration step.
⚠️ Common Interview Pitfalls
  • ✕
    Averaging weights of models trained from completely different random initializations; they occupy distinct loss basins, causing complete prediction collapse
  • ✕
    Evaluating SWA on BatchNorm architectures without updating running statistics, producing terrible benchmark scores
🎯 Interviewer Follow-up Inquiries
  • ?
    Why does linear weight interpolation fail between two models trained from different random initializations?
  • ?
    How does Greedy Model Soups decide which fine-tuned checkpoints to include in the average?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM3-050: The Role of Exponential Moving Average (EMA) of Weights in Training📋 Back to BankNext →M3-052: Label Smoothing: Role, Mathematical Mechanism, and Hidden Costs