Scaling Laws & Emergent Abilities

Power-law loss scaling

A common empirical form for test loss $L$ is:

$$L(N, D) \approx L_\infty + a\,N^{-\alpha} + b\,D^{-\beta},$$

where:

• $N$ is parameter count,
• $D$ is number of training tokens (or examples),
• $\alpha,\beta>0$ are exponents you fit from data,
• $L_\infty$ is the irreducible loss floor for the dataset/model class.

On a log-log plot, $N^{-\alpha}$ and $D^{-\beta}$ look like straight lines. That's why power laws are useful: they extrapolate smoothly.

Compute-optimal allocation (Chinchilla-style rule of thumb)

Very roughly, training compute scales like:

$$C \propto N\cdot D.$$

Substituting $D=C/N$ into the displayed loss does not, by itself, force a near-linear token/model rule. Differentiating the two reducible terms gives

$$D^{\beta} \propto N^{\alpha},$$

up to constants from $a$, $b$, $\alpha$, $\beta$, and the compute model. Equivalently, the fitted power-law toy objective suggests an exponent-dependent frontier:

$$D \propto N^{\alpha/\beta}.$$

The Chinchilla-style statement is the empirical next step, not an algebraic consequence of the generic equation alone: Hoffmann et al. found, for their fitted language-model experiments and compute accounting, that compute-optimal training scales model size and training tokens together along the frontier. The practical lesson is still the same: for a fixed compute budget, do not overscale parameters while starving the run of data. The proportionality constant depends on units, data quality, architecture, optimizer, and training recipe.

"Emergent abilities" as sharp transitions

When you measure a capability with a thresholded metric ("accuracy above 50%", "passes a benchmark"), smooth curves in loss can turn into sharp-looking transitions. The underlying performance can still be continuous.