M5-018M5: NLP & Large Language ModelsScaling Laws & Compute AllocationHard
⚡Flashcard Drill➔Mastery:

Guiding Hyperparameter and Architecture Choices via Scaling Laws

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M5-019 →
📐 Mathematical Definition
L(N,D) fit at small scale→predict at large;μP: transfer hyperparams\mathcal{L}(N,D)\ \text{fit at small scale}\to\text{predict at large};\qquad \mu\text{P}:\ \text{transfer hyperparams}
⚡ One-Sentence Core Answer
Scaling laws guide pre-training hyperparameter tuning by using Maximal Update Parametrization (μ\muP) to transfer optimal learning rates from tiny proxy models directly to hundred-billion-parameter models without expensive hyperparameter sweeps.

📌 Key Takeaways

  • •
    The hyperparameter scaling challenge: running grid searches for learning rate, batch size, and initialization on a 70B+ model costs millions of dollars and is practically impossible
  • •
    Maximal Update Parametrization (μ\muP, Yang et al.): rescales layer initializations and learning rates by width dd, guaranteeing that feature representations update at identical rates regardless of model width
  • •
    Predictable zero-shot transfer: optimal hyperparameters (learning rate, weight decay, Adam ϵ\epsilon) found on a 10M proxy model transfer directly to a 100B model without modification

📐 Mathematical Derivations

Mathematical Mechanism: 1. Standard Parametrization (SP) Breakdown: In standard PyTorch initialization: weights are initialized as W∼N(0,1/din)W \sim \mathcal{N}(0, 1/d_{\text{in}}), and learning rate η\eta is constant across all layers. As width d→∞d \to \infty: - Activations h=Wxh = W x have variance Var(h)=d⋅(1/d)=1\text{Var}(h) = d \cdot (1/d) = 1 (stable). - However, weight updates ΔW=−η∇WL\Delta W = -\eta \nabla_W \mathcal{L} scale as Δh=ΔWx∝ηd\Delta h = \Delta W x \propto \eta d, causing activation updates to explode with width dd. Consequently, the optimal learning rate shrinks as dd grows: η∗(d)∝1/d\eta^*(d) \propto 1/d, preventing hyperparameter transfer across model sizes. 2. Maximal Update Parametrization ($\mu$P) Rules: Rescales parameters and learning rates by width multiplier m=d/dbasem = d / d_{\text{base}}: - Hidden-to-hidden weights WhiddenW_{\text{hidden}}: Init∝1d\text{Init} \propto \frac{1}{\sqrt{d}}, Learning Rate ηhidden=ηbasem\eta_{\text{hidden}} = \frac{\eta_{\text{base}}}{m}. - Output projection WheadW_{\text{head}}: Init∝1d\text{Init} \propto \frac{1}{d}, Learning Rate ηhead=ηbasem2\eta_{\text{head}} = \frac{\eta_{\text{base}}}{m^2}. - Attention logits: scaled by 1d\frac{1}{d} rather than 1d\frac{1}{\sqrt{d}}. Under μ\muP, both forward activation norms and backward update norms remain constant O(1)O(1) as width d→∞d \to \infty, ensuring that the optimal learning rate η∗\eta^* is strictly invariant to model width.

🏭 Production Trade-offs

Deep Dive & Engineering Trade-offs: ① Batch Size Scaling Law: Critical batch size Bcrit(L)B_{\text{crit}}(L) scales inversely with loss: Bcrit∝1/LB_{\text{crit}} \propto 1 / L. Early in training, batch size should be small; as training progresses and loss drops, batch size can expand to thousands of sequences without gradient noise saturation. ② Width vs Depth Allocations: Empirical scaling laws prove that scaling width dmodeld_{\text{model}} and depth NlayersN_{\text{layers}} together (aspect ratio N/d≈constantN / d \approx \text{constant}) is compute-optimal. Extremely deep, narrow models suffer from gradient propagation delay; extremely wide, shallow models suffer from memory bandwidth inefficiencies. ③ Learning Rate Decay Schedules: Cosine decay is standard; WSD (Warmup-Stable-Decay) has emerged as the modern alternative, maintaining a constant high learning rate for 90%90\% of training and decaying only during annealing, enabling continuous training runs without committing to a fixed token horizon. ④ $\mu$P in Production: Adopted by Microsoft (Cerebras-GPT, Phi-1/2) and modern foundation labs, saving millions of dollars in compute by tuning learning rates exclusively on 100M parameter models. ⑤ Interview Strategy: Explain why Standard Parametrization fails as width expands (Δh∝d\Delta h \propto d), write down the μ\muP scaling rules for hidden and output layers, and explain how it enables zero-shot hyperparameter transfer.
⚠️ Common Interview Pitfalls
  • ✕
    Attempting to transfer standard PyTorch learning rates from small models to large models without μ\muP (causes gradient explosion or divergence)
  • ✕
    Scaling depth aggressively without scaling width proportionally
  • ✕
    Locking in a fixed cosine learning rate schedule before knowing the final pre-training token budget (use WSD instead)
🎯 Interviewer Follow-up Inquiries
  • ?
    Why do attention logits in μ\muP require 1/d1/d scaling instead of standard 1/d1/\sqrt{d} scaling as width approaches infinity?
  • ?
    What is the Warmup-Stable-Decay (WSD) schedule, and why does it outperform cosine decay for continuous pre-training?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM5-017: Scaling Strategies Under Data-Constrained Regimes📋 Back to BankNext →M5-019: Post-Training Scaling (RL and Test-Time Compute) vs. Pre-Training Scaling