M5-018M5: NLP & Large Language ModelsScaling Laws & Compute AllocationHard
⚡Flashcard Drill➔Mastery:
Guiding Hyperparameter and Architecture Choices via Scaling Laws
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Scaling laws guide pre-training hyperparameter tuning by using Maximal Update Parametrization (P) to transfer optimal learning rates from tiny proxy models directly to hundred-billion-parameter models without expensive hyperparameter sweeps.
📌 Key Takeaways
- •The hyperparameter scaling challenge: running grid searches for learning rate, batch size, and initialization on a 70B+ model costs millions of dollars and is practically impossible
- •Maximal Update Parametrization (P, Yang et al.): rescales layer initializations and learning rates by width , guaranteeing that feature representations update at identical rates regardless of model width
- •Predictable zero-shot transfer: optimal hyperparameters (learning rate, weight decay, Adam ) found on a 10M proxy model transfer directly to a 100B model without modification
📐 Mathematical Derivations
Mathematical Mechanism: 1. Standard Parametrization (SP) Breakdown: In standard PyTorch initialization: weights are initialized as , and learning rate is constant across all layers. As width : - Activations have variance (stable). - However, weight updates scale as , causing activation updates to explode with width . Consequently, the optimal learning rate shrinks as grows: , preventing hyperparameter transfer across model sizes. 2. Maximal Update Parametrization ($\mu$P) Rules: Rescales parameters and learning rates by width multiplier : - Hidden-to-hidden weights : , Learning Rate . - Output projection : , Learning Rate . - Attention logits: scaled by rather than . Under P, both forward activation norms and backward update norms remain constant as width , ensuring that the optimal learning rate is strictly invariant to model width.
🏭 Production Trade-offs
Deep Dive & Engineering Trade-offs: ① Batch Size Scaling Law: Critical batch size scales inversely with loss: . Early in training, batch size should be small; as training progresses and loss drops, batch size can expand to thousands of sequences without gradient noise saturation. ② Width vs Depth Allocations: Empirical scaling laws prove that scaling width and depth together (aspect ratio ) is compute-optimal. Extremely deep, narrow models suffer from gradient propagation delay; extremely wide, shallow models suffer from memory bandwidth inefficiencies. ③ Learning Rate Decay Schedules: Cosine decay is standard; WSD (Warmup-Stable-Decay) has emerged as the modern alternative, maintaining a constant high learning rate for of training and decaying only during annealing, enabling continuous training runs without committing to a fixed token horizon. ④ $\mu$P in Production: Adopted by Microsoft (Cerebras-GPT, Phi-1/2) and modern foundation labs, saving millions of dollars in compute by tuning learning rates exclusively on 100M parameter models. ⑤ Interview Strategy: Explain why Standard Parametrization fails as width expands (), write down the P scaling rules for hidden and output layers, and explain how it enables zero-shot hyperparameter transfer.
⚠️ Common Interview Pitfalls
- ✕Attempting to transfer standard PyTorch learning rates from small models to large models without P (causes gradient explosion or divergence)
- ✕Scaling depth aggressively without scaling width proportionally
- ✕Locking in a fixed cosine learning rate schedule before knowing the final pre-training token budget (use WSD instead)
🎯 Interviewer Follow-up Inquiries
- ?Why do attention logits in P require scaling instead of standard scaling as width approaches infinity?
- ?What is the Warmup-Stable-Decay (WSD) schedule, and why does it outperform cosine decay for continuous pre-training?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚