M5-014M5: NLP & Large Language ModelsScaling Laws & Compute AllocationEasy
⚡Flashcard Drill➔Mastery:
Core Conclusions of Kaplan vs. Chinchilla Scaling Laws
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Kaplan et al. concluded that model parameters should scale faster than training tokens (), while Chinchilla proved that parameters and tokens should scale in equal proportion (, tokens per parameter).
📌 Key Takeaways
- •Kaplan Scaling Law (OpenAI 2020): concluded that model size matters far more than data; recommended scaling parameters aggressively while scaling data slowly, leading to severely under-trained models like GPT-3 175B (300B tokens)
- •Chinchilla Correction (DeepMind Hoffmann et al. 2022): fixed sub-optimal learning rate schedules in Kaplan's setup; proved parameters and tokens should scale symmetrically ( ratio in compute exponents)
- •Compute-optimal rule: for compute-optimal training, a model should be trained on approximately 20 tokens per parameter (e.g., a 70B model requires tokens for compute optimality)
📐 Mathematical Derivations
Mathematical Mechanism: 1. The Compute Budget Constraint: Pre-training compute FLOPs are approximately: 2. Power-Law Loss Formulation: Loss is modeled as a function of parameters and dataset size : Where is irreducible loss. To find the compute-optimal allocation of and that minimizes subject to : Set up the Lagrangian: . Solving and yields: 3. Kaplan vs Chinchilla Parameter Values: - Kaplan (2020): . Recommended scale faster than . - Chinchilla (2022): Evaluated models where LR schedule cosine duration matched actual token count. Fitted: . Thus, both parameters and tokens should scale in equal proportion: , with ratio .
🏭 Production Trade-offs
Deep Dive & Engineering Trade-offs: ① Kaplan's Flaw: Kaplan kept the cosine learning rate schedule fixed to a large token count while evaluating models on early checkpoints. Because models at early steps of a long cosine schedule have higher learning rates than models trained with a cosine schedule tuned specifically for that step count, Kaplan systematically underestimated the power of training on more data. ② Why Chinchilla 70B Beat Gopher 280B: DeepMind's Chinchilla (70B parameters trained on 1.4T tokens) used the exact same compute budget as Gopher (280B trained on 300B tokens), but decisively outperformed Gopher across all tasks while being smaller and faster at inference. ③ Inference Over-Training Trend: Chinchilla defines training compute optimality. In production, models are used for billions of inference requests; over-training smaller models (e.g., LLaMA-3 8B trained on 15T tokens tokens/param) vastly outperforms Chinchilla optimality in total life-cycle cost. ④ Downstream Benchmark Correlation: Pre-training cross-entropy loss follows strict power-law scaling across 6 orders of magnitude; downstream task performance generally scales monotonically with cross-entropy loss. ⑤ Interview Strategy: Write down , set up the Lagrangian optimization, contrast the Kaplan ( vs ) and Chinchilla ( vs ) exponents, and explain Kaplan's learning rate schedule flaw.
⚠️ Common Interview Pitfalls
- ✕Assuming Chinchilla's 20 tokens/param is a hard upper bound (it is the training compute-optimal point, not the inference cost-optimal point)
- ✕Believing Kaplan's scaling law is still valid (it was superseded by Chinchilla due to experimental methodology flaws)
- ✕Forgetting the factor of 6 in (2 FLOPs per parameter forward + 4 FLOPs backward)
🎯 Interviewer Follow-up Inquiries
- ?Why did Kaplan's fixed cosine learning rate schedule systematically underestimate the benefit of training tokens?
- ?How does factoring in inference query volume alter the optimal token-to-parameter ratio beyond Chinchilla's 20:1?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚