M3-012M3: Deep Learning FoundationsWeight InitializationHard
⚡Flashcard Drill➔Mastery:
Maximal Update Parametrization (μP): Concepts and Hyperparameter Transfer
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
μP adjusts weight initialization and learning rates by layer width, ensuring optimal hyperparameters (learning rate, schedule) remain invariant as model width scales.
📌 Key Takeaways
- •Problem: under Standard Parametrization (SP), optimal learning rate systematically shifts when scaling width from 256 to 8192
- •μP solution: scales hidden initializations by and output layer learning rates by
- •Hyperparameter transfer: tune optimal learning rate, weight decay, and schedules cheaply on a small model and transfer directly to massive models
📐 Mathematical Derivations
Mathematical Foundation (Yang & Hu, 2021; Tensor Programs):
Under Standard Parametrization (SP), hidden weight matrices are initialized with , and updated with uniform learning rate . As width , feature representations either blow up or collapse into the 'neural tangent kernel (NTK)' lazy-training regime where features do not adapt.
μP Rules:
- Hidden Weights ($W_{\text{hid}} \in \mathbb{R}^{n \times n}$): Initialization ; Learning rate .
- Output / Readout Projection ($W_{\text{out}} \in \mathbb{R}^{d_{\text{out}} \times n}$): Initialization ; Learning rate .
- Attention Logits: Scale attention scores by instead of .
Under μP, every layer's forward activation change and backward gradient update remain in the infinite-width limit. As a result, the optimal learning rate is independent of width .
Under Standard Parametrization (SP), hidden weight matrices are initialized with , and updated with uniform learning rate . As width , feature representations either blow up or collapse into the 'neural tangent kernel (NTK)' lazy-training regime where features do not adapt.
μP Rules:
- Hidden Weights ($W_{\text{hid}} \in \mathbb{R}^{n \times n}$): Initialization ; Learning rate .
- Output / Readout Projection ($W_{\text{out}} \in \mathbb{R}^{d_{\text{out}} \times n}$): Initialization ; Learning rate .
- Attention Logits: Scale attention scores by instead of .
Under μP, every layer's forward activation change and backward gradient update remain in the infinite-width limit. As a result, the optimal learning rate is independent of width .
🏭 Production Trade-offs
Industrial value: Training multi-billion parameter models is too expensive for exhaustive grid searches. μP allows teams to perform comprehensive hyperparameter searches on small proxy models (e.g., 100M parameters) and transfer the discovered optimal learning rate directly to 70B+ production runs without re-tuning.
⚠️ Common Interview Pitfalls
- ✕Assuming μP enables hyperparameter transfer across depth; μP strictly governs width scaling (depth scaling requires additional depth-μP rules)
- ✕Applying μP without scaling the output readout learning rate by , which breaks the parameterization stability
🎯 Interviewer Follow-up Inquiries
- ?Why must attention logits be scaled by rather than in the infinite-width μP regime?
- ?What is the difference between feature learning and the Neural Tangent Kernel (NTK) regime when scaling width?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚