M5-014M5: NLP & Large Language ModelsScaling Laws & Compute AllocationEasy
⚡Flashcard Drill➔Mastery:

Core Conclusions of Kaplan vs. Chinchilla Scaling Laws

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M5-015 →
📐 Mathematical Definition
L(N,D)=L∞+AN−α+BD−β;Chinchilla:D⋆≈20N⋆\mathcal{L}(N,D)=\mathcal{L}_\infty+AN^{-\alpha}+BD^{-\beta};\qquad \text{Chinchilla}: D^{\star}\approx20N^{\star}
⚡ One-Sentence Core Answer
Kaplan et al. concluded that model parameters should scale faster than training tokens (N∝C0.73,D∝C0.27N \propto C^{0.73}, D \propto C^{0.27}), while Chinchilla proved that parameters and tokens should scale in equal proportion (N∝C0.5,D∝C0.5N \propto C^{0.5}, D \propto C^{0.5}, ≈20\approx 20 tokens per parameter).

📌 Key Takeaways

  • •
    Kaplan Scaling Law (OpenAI 2020): concluded that model size matters far more than data; recommended scaling parameters aggressively while scaling data slowly, leading to severely under-trained models like GPT-3 175B (300B tokens)
  • •
    Chinchilla Correction (DeepMind Hoffmann et al. 2022): fixed sub-optimal learning rate schedules in Kaplan's setup; proved parameters NN and tokens DD should scale symmetrically (1:11:1 ratio in compute exponents)
  • •
    Compute-optimal rule: for compute-optimal training, a model should be trained on approximately 20 tokens per parameter (e.g., a 70B model requires ≈1.4T\approx 1.4\text{T} tokens for compute optimality)

📐 Mathematical Derivations

Mathematical Mechanism: 1. The Compute Budget Constraint: Pre-training compute FLOPs are approximately: C≈6NDC \approx 6 N D 2. Power-Law Loss Formulation: Loss is modeled as a function of parameters NN and dataset size DD: L(N,D)=E+ANα+BDβL(N, D) = E + \frac{A}{N^\alpha} + \frac{B}{D^\beta} Where EE is irreducible loss. To find the compute-optimal allocation of N(C)N(C) and D(C)D(C) that minimizes L(N,D)L(N, D) subject to 6ND=C6 N D = C: Set up the Lagrangian: L(N,D,λ)=E+AN−α+BD−β+λ(6ND−C)\mathcal{L}(N, D, \lambda) = E + A N^{-\alpha} + B D^{-\beta} + \lambda (6 N D - C). Solving ∂L∂N=0\frac{\partial \mathcal{L}}{\partial N} = 0 and ∂L∂D=0\frac{\partial \mathcal{L}}{\partial D} = 0 yields: N∝Cβα+β,D∝Cαα+βN \propto C^{\frac{\beta}{\alpha + \beta}}, \quad D \propto C^{\frac{\alpha}{\alpha + \beta}} 3. Kaplan vs Chinchilla Parameter Values: - Kaplan (2020): α=0.076,β=0.057  ⟹  N∝C0.73,D∝C0.27\alpha = 0.076, \beta = 0.057 \implies N \propto C^{0.73}, D \propto C^{0.27}. Recommended NN scale 3×3\times faster than DD. - Chinchilla (2022): Evaluated >400>400 models where LR schedule cosine duration matched actual token count. Fitted: α≈0.34,β≈0.28  ⟹  N∝C0.50,D∝C0.50\alpha \approx 0.34, \beta \approx 0.28 \implies N \propto C^{0.50}, D \propto C^{0.50}. Thus, both parameters and tokens should scale in equal proportion: Nopt∝C,Dopt∝CN_{\text{opt}} \propto \sqrt{C}, D_{\text{opt}} \propto \sqrt{C}, with ratio D/N≈20D / N \approx 20.

🏭 Production Trade-offs

Deep Dive & Engineering Trade-offs: ① Kaplan's Flaw: Kaplan kept the cosine learning rate schedule fixed to a large token count while evaluating models on early checkpoints. Because models at early steps of a long cosine schedule have higher learning rates than models trained with a cosine schedule tuned specifically for that step count, Kaplan systematically underestimated the power of training on more data. ② Why Chinchilla 70B Beat Gopher 280B: DeepMind's Chinchilla (70B parameters trained on 1.4T tokens) used the exact same compute budget as Gopher (280B trained on 300B tokens), but decisively outperformed Gopher across all tasks while being 4×4\times smaller and faster at inference. ③ Inference Over-Training Trend: Chinchilla defines training compute optimality. In production, models are used for billions of inference requests; over-training smaller models (e.g., LLaMA-3 8B trained on 15T tokens ≈1875\approx 1875 tokens/param) vastly outperforms Chinchilla optimality in total life-cycle cost. ④ Downstream Benchmark Correlation: Pre-training cross-entropy loss follows strict power-law scaling across 6 orders of magnitude; downstream task performance generally scales monotonically with cross-entropy loss. ⑤ Interview Strategy: Write down C≈6NDC \approx 6ND, set up the Lagrangian optimization, contrast the Kaplan (C0.73C^{0.73} vs C0.27C^{0.27}) and Chinchilla (C0.5C^{0.5} vs C0.5C^{0.5}) exponents, and explain Kaplan's learning rate schedule flaw.
⚠️ Common Interview Pitfalls
  • ✕
    Assuming Chinchilla's 20 tokens/param is a hard upper bound (it is the training compute-optimal point, not the inference cost-optimal point)
  • ✕
    Believing Kaplan's scaling law is still valid (it was superseded by Chinchilla due to experimental methodology flaws)
  • ✕
    Forgetting the factor of 6 in C=6NDC = 6ND (2 FLOPs per parameter forward + 4 FLOPs backward)
🎯 Interviewer Follow-up Inquiries
  • ?
    Why did Kaplan's fixed cosine learning rate schedule systematically underestimate the benefit of training tokens?
  • ?
    How does factoring in inference query volume alter the optimal token-to-parameter ratio beyond Chinchilla's 20:1?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM5-013: Pre-training Domain Mixtures and the Role of the Annealing Phase📋 Back to BankNext →M5-015: Why Inference Costs Alter Compute-Optimal Scaling Choices