Chinchilla Compute-Optimal Scaling Laws

The Chinchilla Compute-Optimal Scaling Laws (Hoffmann et al., DeepMind) formalizes the exact mathematical Pareto frontier allocating compute budgets between model parameters $N$ and dataset tokens $D$; fitting the empirical power-law loss surface $L(N, D) = E + \frac{A}{N^\alpha} + \frac{B}{D^\beta}$ (where $\alpha \approx 0.34, \beta \approx 0.28$) under fixed training compute budget $C \approx 6ND$ using Lagrange Multipliers yields the optimal scaling trajectory: $N_{\text{opt}} \propto C^{\frac{\beta}{\alpha+\beta}}$ and $D_{\text{opt}} \propto C^{\frac{\alpha}{\alpha+\beta}}$; this formally proves that parameters and tokens should scale in near-equal proportions, establishing the foundational rule of thumb: optimal pre-training requires $\approx 20$ tokens per model parameter ($D/N \approx 20$).

Multi-million dollar pre-training compute budgeting, foundation model capacity planning, and scaling law research.

Early architectures (GPT-3 175B on 300B tokens) were severely under-trained; Chinchilla guided modern over-trained dense models (Llama 7B on 2T+ tokens) yielding vastly superior downstream inference efficiency.

Test Your Knowledge: Practice Questions for "Chinchilla Compute-Optimal Scaling Laws"

Single choice pitfall questions with instant feedback and mistake tracking.