M3-012M3: Deep Learning FoundationsWeight InitializationHard
⚡Flashcard Drill➔Mastery:

Maximal Update Parametrization (μP): Concepts and Hyperparameter Transfer

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M3-013 →
📐 Mathematical Definition
η∝1/width (output layers)\eta\propto 1/\text{width}\ (\text{output layers})
⚡ One-Sentence Core Answer
μP adjusts weight initialization and learning rates by layer width, ensuring optimal hyperparameters (learning rate, schedule) remain invariant as model width scales.

📌 Key Takeaways

  • •
    Problem: under Standard Parametrization (SP), optimal learning rate systematically shifts when scaling width from 256 to 8192
  • •
    μP solution: scales hidden initializations by 1/width1/\text{width} and output layer learning rates by 1/width1/\text{width}
  • •
    Hyperparameter transfer: tune optimal learning rate, weight decay, and schedules cheaply on a small model and transfer directly to massive models

📐 Mathematical Derivations

Mathematical Foundation (Yang & Hu, 2021; Tensor Programs):
Under Standard Parametrization (SP), hidden weight matrices are initialized with Var(W)∼1/n\text{Var}(W) \sim 1/n, and updated with uniform learning rate η\eta. As width n→∞n \to \infty, feature representations either blow up or collapse into the 'neural tangent kernel (NTK)' lazy-training regime where features do not adapt.
μP Rules:
- Hidden Weights ($W_{\text{hid}} \in \mathbb{R}^{n \times n}$): Initialization Var(W)∼1/n\text{Var}(W) \sim 1/n; Learning rate ηhid∼Θ(1)\eta_{\text{hid}} \sim \Theta(1).
- Output / Readout Projection ($W_{\text{out}} \in \mathbb{R}^{d_{\text{out}} \times n}$): Initialization Var(W)∼1/n2\text{Var}(W) \sim 1/n^2; Learning rate ηout∼Θ(1/n)\eta_{\text{out}} \sim \Theta(1/n).
- Attention Logits: Scale attention scores by 1/d1/d instead of 1/d1/\sqrt{d}.
Under μP, every layer's forward activation change Δh\Delta h and backward gradient update remain O(1)O(1) in the infinite-width limit. As a result, the optimal learning rate is independent of width nn.

🏭 Production Trade-offs

Industrial value: Training multi-billion parameter models is too expensive for exhaustive grid searches. μP allows teams to perform comprehensive hyperparameter searches on small proxy models (e.g., 100M parameters) and transfer the discovered optimal learning rate directly to 70B+ production runs without re-tuning.
⚠️ Common Interview Pitfalls
  • ✕
    Assuming μP enables hyperparameter transfer across depth; μP strictly governs width scaling (depth scaling requires additional depth-μP rules)
  • ✕
    Applying μP without scaling the output readout learning rate by 1/n1/n, which breaks the parameterization stability
🎯 Interviewer Follow-up Inquiries
  • ?
    Why must attention logits be scaled by 1/d1/d rather than 1/d1/\sqrt{d} in the infinite-width μP regime?
  • ?
    What is the difference between feature learning and the Neural Tangent Kernel (NTK) regime when scaling width?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM3-011: Common Initialization Techniques in Transformers📋 Back to BankNext →M3-013: Comparing Sigmoid, Tanh, and ReLU: Mathematical Properties and Trade-offs