Chinchilla Compute-Optimal Scaling Laws

The Chinchilla Compute-Optimal Scaling Laws (Hoffmann et al., DeepMind) formalizes the exact mathematical Pareto frontier allocating compute budgets between model parameters $N$ and dataset tokens $D$; fitting the empirical power-law loss surface $L(N, D) = E + \frac{A}{N^\alpha} + \frac{B}{D^\beta}$ (where $\alpha \approx 0.34, \beta \approx 0.28$) under fixed training compute budget $C \approx 6ND$ using Lagrange Multipliers yields the optimal scaling trajectory: $N_{\text{opt}} \propto C^{\frac{\beta}{\alpha+\beta}}$ and $D_{\text{opt}} \propto C^{\frac{\alpha}{\alpha+\beta}}$; this formally proves that parameters and tokens should scale in near-equal proportions, establishing the foundational rule of thumb: optimal pre-training requires $\approx 20$ tokens per model parameter ($D/N \approx 20$).

Multi-million dollar pre-training compute budgeting, foundation model capacity planning, and scaling law research.

Early architectures (GPT-3 175B on 300B tokens) were severely under-trained; Chinchilla guided modern over-trained dense models (Llama 7B on 2T+ tokens) yielding vastly superior downstream inference efficiency.

Derive the optimal scaling exponents $a = \frac{\beta}{\alpha+\beta}$ and $b = \frac{\alpha}{\alpha+\beta}$ using Lagrange Multipliers under compute constraint $C = 6ND$?

Explain the mathematical derivation showing why training compute is precisely $6N$ FLOPs per token (2N forward + 4N backward)?

Why does industry embrace Inference-Aware Over-training (e.g. 8B models trained on 15T tokens) to optimize lifetime serving cost?

Analyze the mathematical discontinuity between smooth cross-entropy scaling and sudden emergent metric jumps on non-linear accuracy thresholds?

How does high-quality synthetic data generation and reasoning search bypass the impending internet human text data wall?