Scaling Laws

Scaling laws state that the cross-entropy loss follows power laws in parameters N, data D and compute C: $L(N) \propto N^{-\alpha}$, $L(D) \propto D^{-\beta}$, $L(C) \propto C^{-\gamma}$, with α≈0.076 and β≈0.095. Kaplan's law argued parameters matter more; the Chinchilla law (2022) corrected this: data and parameters should scale together, with optimal data ≈ 20× parameters — a 70B model needs ~1.4T tokens, hence 'data beats parameters'. The LLaMA family was trained at exactly this ratio.

picking model size and token budget, estimating training cost, extrapolating performance and choosing architecture.

the optimal (N, D) split under a fixed compute budget. Early models like GPT-3 were heavily undertrained; Chinchilla showed data gets more return per FLOP at fixed compute. Power laws also enable extrapolation: runs at 1e20 FLOPs predict loss at 1e24 FLOPs within ~5% error, justifying the 'experiment small, scale up' engineering route.

Test Your Knowledge: Practice Questions for "Scaling Laws"

Single choice pitfall questions with instant feedback and mistake tracking.