Scaling Laws

Scaling laws state that the cross-entropy loss follows power laws in parameters N, data D and compute C: $L(N) \propto N^{-\alpha}$, $L(D) \propto D^{-\beta}$, $L(C) \propto C^{-\gamma}$, with α≈0.076 and β≈0.095. Kaplan's law argued parameters matter more; the Chinchilla law (2022) corrected this: data and parameters should scale together, with optimal data ≈ 20× parameters — a 70B model needs ~1.4T tokens, hence 'data beats parameters'. The LLaMA family was trained at exactly this ratio.

picking model size and token budget, estimating training cost, extrapolating performance and choosing architecture.

the optimal (N, D) split under a fixed compute budget. Early models like GPT-3 were heavily undertrained; Chinchilla showed data gets more return per FLOP at fixed compute. Power laws also enable extrapolation: runs at 1e20 FLOPs predict loss at 1e24 FLOPs within ~5% error, justifying the 'experiment small, scale up' engineering route.

Power-law forms? Kaplan vs Chinchilla differences?

Why does data matter more than parameters at fixed compute?

Chinchilla's content and how the 20:1 data-to-param ratio was derived?

Do scaling laws extrapolate forever? Saturation and emergence?

Using scaling laws to choose N/D and estimate training cost?