02 - LLM scaling laws and emergent capabilities

You have a fixed compute budget and must decide how to spend it: a larger model trained for longer on little data, or a smaller one trained on a lot? Before 2020 the answer was an intuition; today it is an optimization problem with a closed-form solution, provided you accept that the error of a language model follows regular power laws in three quantities: parameters, data and compute. This article builds those laws on a miniature experiment where the true answer is known, derives the allocation rule of Hoffmann and colleagues (2022) and checks it numerically, and then looks, with a toy model, at why some capabilities seem to appear all of a sudden. No language model is trained: what is measured is how to reason about scaling charts without trusting their extremes.

The listings were run twice with identical output with numpy 2.4.6 and scipy 1.17.1, on synthetic data with a fixed seed. The figures on real models come only from the three scientific papers listed in the sources, and where I reproduced them I say so.

What a scaling law is

Kaplan and colleagues (2020) measure the cross-entropy loss of language Transformers as the sizes change and report that the loss follows a power law in the size of the model, in that of the dataset and in the compute, with trends spanning more than seven orders of magnitude. For the number N of non-embedding parameters alone, training to convergence on sufficient data, they write L(N) = (N_c/N)^α_N with α_N ≈ 0.076; for the data D alone, with α_D ≈ 0.095 (section 1.2). An exponent this small means that to halve the reducible part of the error you must multiply the size by 2^1/0.076, about 9000 times: power laws promise regular improvements, not cheap ones.

Three caveats hold for every scaling law. It is an empirical fit to a family of models, a set of data and a range of scales: outside the range it is an extrapolation. It describes the loss, not a specific capability: the link between a loss and what the model can do has to be measured separately. Finally it almost always contains a floor, an error that no amount of parameters or data removes, and how that floor is treated changes the estimated exponents.

A power law in miniature, with a known answer

To see a scaling law without training a language model you need a problem where the truth is known. The listing uses a linear regression with 4000 independent Gaussian features and true coefficients w_k = k^-1: the high-index features matter less and less. A "model of size m" uses only the first m features. The expected error on new data has a closed form: the sum of the w_k² of the excluded features (their unexplained variance) plus the measurement noise σ² = 0.001. As a second, independent calculation, least squares is actually fitted on 8000 examples and the error is measured on another 8000. The sum of the squares of the excluded coefficients is about 1/m, so I expect a power law with an exponent close to 1 on the reducible part alone.

import numpy as np
from scipy.optimize import curve_fit

rng = np.random.default_rng(0)
D, a = 4000, 1.0                       # 4000 features; true coefficients w_k = k^-1
w = np.arange(1, D + 1, dtype=float) ** -a
sigma2 = 0.001                         # measurement noise: the floor no model removes
n = 8000

X = rng.standard_normal((n, D))
y = X @ w + np.sqrt(sigma2) * rng.standard_normal(n)
Xt = rng.standard_normal((n, D))
yt = Xt @ w + np.sqrt(sigma2) * rng.standard_normal(n)

ms = np.array([4, 8, 16, 32, 64, 128, 256])
chiusa, misurata = [], []
print(f"{'m':>4} {'error (closed form)':>22} {'measured error':>16}")
for m in ms:
    tail = (w[m:] ** 2).sum()                          # variance of the excluded features
    chiusa.append(tail + sigma2)                       # formula: tail + noise (ideal model)
    coef, *_ = np.linalg.lstsq(X[:, :m], y, rcond=None)
    misurata.append(np.mean((yt - Xt[:, :m] @ coef) ** 2))
    print(f"{m:>4} {chiusa[-1]:>22.5f} {misurata[-1]:>16.5f}")

chiusa, misurata = np.array(chiusa), np.array(misurata)
red = chiusa - sigma2
print("\nlog-log slope of the reducible part only:", round(np.polyfit(np.log(ms), np.log(red), 1)[0], 3),
      "- expected 1-2a =", 1 - 2 * a)
print("log-log slope of the total error (with the floor):", round(np.polyfit(np.log(ms), np.log(misurata), 1)[0], 3))
f = lambda m, c, A, al: c + A * m ** -al
(c, A, al), _ = curve_fit(f, ms.astype(float), misurata, p0=[0.001, 1, 1.0])
print(f"fit of c + A*m^-alpha on the measurements: c={c:.4f} (true {sigma2}), alpha={al:.3f} (expected {2 * a - 1})")

# Output:
#    m    error (closed form)   measured error
#    4                0.22207          0.22523
#    8                0.11826          0.12097
#   16                0.06134          0.06283
#   32                0.03152          0.03150
#   64                0.01625          0.01639
#  128                0.00853          0.00862
#  256                0.00465          0.00477
#
# log-log slope of the reducible part only: -0.988 - expected 1-2a = -1.0
# log-log slope of the total error (with the floor): -0.937
# fit of c + A*m^-alpha on the measurements: c=-0.0014 (true 0.001), alpha=0.909 (expected 1.0)

Closed form and measurement coincide within 3%: the excess that remains is explained by the estimation error, because 8000 examples are not infinite (about m/n, 3% at m = 256). The slope on the log-log plane of the reducible part alone is −0.988, close to −1; but that of the total error, which includes the floor of 0.001, is already −0.937: the floor flattens the curve at large scales. The most instructive result is the last line. The attempt to fit c + A·m^-α to the seven measurements alone returns a negative floor (−0.0014, against 0.001 true) and α = 0.909 instead of 1: with few points and a little noise, the floor and the exponent compensate each other and the fit does not separate them. This is why published exponents have confidence intervals, and why extrapolating by many orders of magnitude from a single fit is fragile.

How to spend a budget: Chinchilla

Hoffmann and colleagues (2022) propose describing the final loss of a model with N parameters trained on D tokens as L(N, D) = E + A/N^α + B/D^β (section 3.3). The first term is the entropy of the text, the second the limit of a model of finite size, the third the fact of not training to convergence. Substituting the compute constraint C ≈ 6ND, which the authors take from Kaplan and colleagues, gives an optimal size N_opt = G·(C/6)^a and D_opt = G^-1·(C/6)^b, with a = β/(α+β), b = α/(α+β) and G = (αA/βB)^1/(α+β). The constants reported in appendix D.2 are E = 1.69, A = 406.4, B = 410.7, α = 0.34, β = 0.28.

The listing uses those constants. For four budgets it compares the closed form with a brute-force minimization over 400,001 values of N (with D = C/6N), that is a second calculation that does not know the algebra. Then it computes the loss predicted by the formula for two configurations from table 1 of the paper: Gopher (280 billion parameters, 300 billion tokens) and Chinchilla (70 billion, 1.4 trillion tokens).

import numpy as np

# constants reported in section D.2 of Hoffmann et al. (2022)
E, A, B, alpha, beta = 1.69, 406.4, 410.7, 0.34, 0.28
L = lambda N, D: E + A / N ** alpha + B / D ** beta

G = (alpha * A / (beta * B)) ** (1 / (alpha + beta))
a, b = beta / (alpha + beta), alpha / (alpha + beta)
print(f"exponents: a = beta/(alpha+beta) = {a:.3f}, b = alpha/(alpha+beta) = {b:.3f}")

print(f"{'C (FLOP)':>9} | {'N closed':>10} {'N grid':>10} | {'D closed':>10} {'D/N':>6} | {'L':>6}")
for C in (1e18, 1e20, 5.76e23, 1e26):
    N_c = G * (C / 6) ** a                    # closed form, equation (4) of the paper
    D_c = (C / 6) / N_c
    N = np.logspace(6, 14, 400001)            # fine grid: I minimize L(N, C/(6N)) by brute force
    N_g = N[np.argmin(L(N, C / (6 * N)))]
    print(f"{C:>9.2e} | {N_c:>10.3e} {N_g:>10.3e} | {D_c:>10.3e} {D_c / N_c:>6.1f} | {L(N_c, D_c):>6.3f}")

print("\nComparison at equal compute, according to the formula:")
for nome, N, D in (("280B on 300B tokens", 280e9, 300e9), ("70B on 1.4T tokens", 70e9, 1.4e12)):
    print(f"{nome:>20}: C = 6ND = {6 * N * D:.2e} FLOP, L = {L(N, D):.3f}")

# Output:
# exponents: a = beta/(alpha+beta) = 0.452, b = alpha/(alpha+beta) = 0.548
#  C (FLOP) |   N closed     N grid |   D closed    D/N |      L
#  1.00e+18 |  8.058e+07  8.058e+07 |  2.068e+09   25.7 |  3.535
#  1.00e+20 |  6.449e+08  6.449e+08 |  2.585e+10   40.1 |  2.600
#  5.76e+23 |  3.219e+10  3.219e+10 |  2.982e+12   92.6 |  1.931
#  1.00e+26 |  3.305e+11  3.305e+11 |  5.043e+13  152.6 |  1.799
#
# Comparison at equal compute, according to the formula:
#  280B on 300B tokens: C = 6ND = 5.04e+23 FLOP, L = 1.993
#   70B on 1.4T tokens: C = 6ND = 5.88e+23 FLOP, L = 1.937

The closed form and the brute-force search give the same optimum to four digits, so the algebra is right. The ratio between tokens and parameters at the optimum is not constant: it rises from 26 to 153 with the budget, because the exponents a and b are not equal (0.452 and 0.548 with the rounded constants; the paper reports 0.46 and 0.54). The paper summarizes the result by saying that for every doubling of the model size the training tokens should double too, that is exponents close to 0.5 in all three approaches it uses. Two caveats about my numbers. The first: at 5.76·10²³ FLOP, the Gopher budget stated in the paper, the formula with the rounded constants gives 32 billion parameters, while the caption of figure 4 of the paper indicates 40 billion: the optimum is sensitive to the constants, and rounding them to two digits is not enough to reproduce the published figure.

The second concerns the compute: 6ND is an approximation: for the two models it gives 5.04 and 5.88·10²³ FLOP, while the paper treats them as equal compute with a more detailed count (appendix F). With the formula Chinchilla has a lower predicted loss (1.937 against 1.993), consistent with what the paper reports having measured for them; the numbers in the table are those of the model and not a measurement.

The practical message is another one. Kaplan and colleagues (2020) concluded that compute-optimal training consists of training very large models on a relatively modest amount of data, stopping well before convergence; Hoffmann and colleagues, with more than 400 models from 70 million to over 16 billion parameters on 5-500 billion tokens, find that size and data should be scaled together, and point to the 70B model trained on 1.4 trillion tokens as an example that at equal compute outperforms the larger models of that comparison. The two conclusions are different, and this article has not verified what the difference depends on: it is a good reason not to treat "the law" as a constant of nature.

"Emergent" capabilities and the yardstick you measure them with

People speak of emergent capabilities when an ability seems absent in small models and suddenly present in large ones, and unpredictable from the smaller scales. Schaeffer and colleagues (2023) propose an alternative explanation: for a task and a family of models, the apparent jump depends on the choice of metric, because nonlinear or discontinuous metrics distort a per-token error that falls smoothly. In their example the exact match on a sequence of k tokens requires all of them to be right. The miniature scheme is easy to reproduce: the per-token error falls like a power law (err = 0.5·(N/10⁶)^-0.2, an exponent chosen by me, with no link to any real model); the discrete metric is the probability (1−err)¹⁰ that ten tokens are all right; the continuous metric is the expected number of wrong tokens, 10·err. The second calculation is a simulation of 2000 sequences per scale.

import numpy as np

rng = np.random.default_rng(0)
k = 10                                   # the task requires all 10 tokens to be right
N = 10.0 ** np.arange(6, 12.5, 0.5)      # model size, from 1e6 to 1e12
err = 0.5 * (N / 1e6) ** -0.2            # error per single token: falls smoothly with N
p = 1 - err                              # probability that a token is right

exact = p ** k                           # discrete metric: the sequence is right only if all tokens are
edit = k * err                           # continuous metric: expected number of wrong tokens

items = 2000                             # second calculation: simulation with 2000 sequences per scale
tok = rng.random((len(N), items, k)) < p[:, None, None]
exact_sim = tok.all(axis=2).mean(axis=1)
edit_sim = (~tok).sum(axis=2).mean(axis=1)

print(f"{'N':>8} {'err/token':>9} | {'exact (formula)':>15} {'exact (sim.)':>13} | {'expected errors':>13} {'(sim.)':>7}")
for i in range(len(N)):
    print(f"{N[i]:>8.1e} {err[i]:>9.3f} | {exact[i]:>15.4f} {exact_sim[i]:>13.4f} | {edit[i]:>13.3f} {edit_sim[i]:>7.3f}")

z = (exact_sim - exact) / np.sqrt(exact * (1 - exact) / items)
print(f"\nmax gap simulation-formula: {np.abs(z).max():.2f} standard errors (expected: below 3 over 13 points)")

# Output:
#        N err/token | exact (formula)  exact (sim.) | expected errors  (sim.)
#  1.0e+06     0.500 |          0.0010        0.0015 |         5.000   5.034
#  3.2e+06     0.397 |          0.0063        0.0015 |         3.972   3.983
#  1.0e+07     0.315 |          0.0226        0.0255 |         3.155   3.170
#  3.2e+07     0.251 |          0.0559        0.0650 |         2.506   2.461
#  1.0e+08     0.199 |          0.1087        0.1100 |         1.991   1.968
#  3.2e+08     0.158 |          0.1789        0.1875 |         1.581   1.575
#  1.0e+09     0.126 |          0.2613        0.2655 |         1.256   1.246
#  3.2e+09     0.100 |          0.3496        0.3555 |         0.998   0.963
#  1.0e+10     0.079 |          0.4380        0.4365 |         0.792   0.786
#  3.2e+10     0.063 |          0.5220        0.5360 |         0.629   0.602
#  1.0e+11     0.050 |          0.5987        0.5895 |         0.500   0.511
#  3.2e+11     0.040 |          0.6668        0.6675 |         0.397   0.413
#  1.0e+12     0.032 |          0.7257        0.7185 |         0.315   0.323
#
# max gap simulation-formula: 2.73 standard errors (expected: below 3 over 13 points)

The continuous metric falls steadily from 5 to 0.3 expected errors and the simulation follows it to within a few hundredths. The exact match, instead, starts at 0.001 at 10⁶ parameters, stays below 0.03 up to 10⁷ and then rises quickly: it looks like an ability that "appears" between 10⁸ and 10¹⁰, although the per-token error falls by a factor of 1.26 for every half order of magnitude, with no jump at all. The comparison between formula and simulation stays below three standard errors over thirteen points. This does not prove that every emergent capability is an artifact: it proves that a jump in a discrete metric is not, by itself, proof of a discontinuity of the model. The same authors write in the abstract that the presumed abilities vanish with different metrics or with better statistics and that they may not be a fundamental property of scale: they argue this on some model families and certain tasks, not on all of them, and the hypothesis remains a hypothesis.

What can and cannot be said about a scaling law

To recap the three experiences, a scaling law is a good way of planning and a bad way of predicting. It plans well because it turns a budget into a choice between parameters and data with a verifiable calculation, as in the second listing. It predicts badly outside its own range, because the constants are sensitive and the parameters of a fit compensate each other, as in the first. And it says little about the task: the loss falls regularly, but what a user needs is a task metric, and how that metric is distorted depends on how it is defined, as in the third.

What you can do now

You can recognize a power law on a log-log chart and you know that the error floor and the exponent compensate each other when the points are few. You can derive and numerically check the optimal allocation of a budget according to the parametric form of Hoffmann and colleagues, and you know how sensitive it is to the constants. You can explain, with an experiment you reproduce in a second, why a jump in a discrete metric does not by itself prove a discontinuity. The next article moves from the model to the knowledge the model does not have: 03 - Fundamental RAG. If you need the architecture first, start from 01 - Transformer Architecture; the generalization error of a classical model is covered in 04 - Bias-Variance Tradeoff.

Later in the series you will also find 07 - Reasoning models: o1, DeepSeek-R1, RLHF, on compute spent at answer time, and 08 - On-prem inference with Ollama, vLLM, llama.cpp, on where that compute runs and how much memory it needs.