Scaling Laws & Emergent Abilities

Power-law loss scaling

A common empirical form for test loss $L$ is:

where:

• $N$ is the number of parameters,
• $D$ is the number of training tokens, each seen once,
• $\alpha,\beta>0$ are exponents fitted to data,
• $L_\infty$ is the loss the fit predicts as $N$ and $D$ both grow without bound; Hoffmann et al. (2022) interpret it as the entropy of natural text.

On a log–log plot each term, such as $a\,N^{-\alpha}$, is a straight line with slope $-\alpha$. That is why power laws are convenient to fit and to extrapolate, although nothing guarantees that the fit holds far beyond the range of the runs it came from.

Compute-optimal allocation (Chinchilla)

Training a dense transformer costs about six floating-point operations per parameter per token, two for the forward pass and four for the backward pass (Kaplan et al., 2020, §2.1):

Substitute $D = C/(6N)$ into the loss and set $dL/dN = 0$. The minimum is at

The two exponents add to 1, so a larger budget is split between a larger model and more tokens. When $\alpha\approx\beta$ both are about $1/2$: parameters and tokens grow at the same rate. That is Hoffmann et al.'s (2022) central finding. Their three estimation methods gave $N^\star\propto C^{0.50}$, $C^{0.49}$ and $C^{0.46}$, against $C^{0.73}$ from Kaplan et al. (2020). With the published, rounded coefficients of their third method ($\alpha=0.34$, $\beta=0.28$), $N^\star\propto C^{0.45}$ (their Table 2 reports $0.46$ for this method) and $D^\star\propto C^{0.55}$, so $D^\star\propto (N^\star)^{1.21}$ and the number of tokens per parameter rises slowly with the budget.

What goes wrong otherwise. At a fixed budget, a model that is too large for its number of tokens is undertrained: the same compute spent on a smaller model trained on more tokens reaches a lower loss. Hoffmann et al. trained Chinchilla (70 billion parameters, 1.4 trillion tokens) with the same compute as Gopher (280 billion parameters, 300 billion tokens), about $5.76\times10^{23}$ FLOPs, and Chinchilla outperformed Gopher on a large range of downstream evaluation tasks.

The fit assumes fresh data. $D$ counts tokens seen once. When unique data runs out, repeating it helps less: Muennighoff et al. (2023) found that up to about four epochs of repeated data were nearly as useful as new data, and that the value of further repetition fell towards zero.

Fitted coefficients are fragile. Besiroglu et al. (2024) refitted the third method to data they extracted from Hoffmann et al.'s Figure 4 and obtained $L_\infty=1.82$, $a=482.01$, $b=2085.43$, $\alpha=0.348$, $\beta=0.366$; they also report that the published coefficients fit the extracted data poorly. Their fit implies about 20 tokens per parameter, in line with Hoffmann et al.'s first two methods. The published coefficients used in the figure below imply about 50 tokens per parameter at $10^{21}$ FLOPs, and more at larger budgets.

"Emergent abilities" and how they are measured

Wei et al. (2022) call an ability emergent if it is absent in smaller models and present in larger ones, and they document many few-shot tasks where accuracy stays near chance and then rises sharply with scale. Schaeffer et al. (2023) argue that many of these jumps come from the metric rather than the model. Exact-match accuracy on a ten-token answer requires all ten tokens to be right, so if token correctness is independent and each token is right with probability $p$, the expected score is $p^{10}$. As $p$ rises steadily from $0.70$ to $0.90$ and then $0.95$, exact match goes from $0.03$ to $0.35$ and then $0.60$. Scored on the same model outputs with a continuous metric, such as token edit distance, many of the reported jumps become smooth. Whether some abilities are emergent in a stronger sense is still debated.