Predicting capability from compute

Loss falls predictably as you add compute, data and parameters, and it does so across many orders of magnitude. That regularity is what justifies spending tens of millions on a training run before anyone knows what the model will be able to do. This section covers the relationship, how it settles the trade-off between model size and data volume for a fixed budget, and its limits — it predicts loss, not capability, which is where the argument over emergent abilities sits.

The relationship#

Loss falls as a power law in compute, data and parameters, holding across many orders of magnitude. It is an empirical regularity rather than a theorem, and it is remarkably consistent.

Say what a power law actually claims, because the phrase is often used to mean no more than 'it goes down'. The form fitted is a floor plus a decaying term. Written for parameters, it is L(N) = E + A times N to the power of minus alpha, where L is the loss, N is the number of parameters, E is the loss you would reach with an infinitely large model, and alpha sets how fast you approach it. The important word is E. Text has an irreducible unpredictability, and no amount of compute removes it, so what falls as a power law is the excess over E rather than the loss itself. A paper reporting a smooth straight line on a log-log plot has usually subtracted an E it also fitted.

The consequence for reading such a plot is that a fixed factor of improvement gets more expensive each time. If alpha is 0.5, halving the excess costs four times the parameters. If alpha is 0.3, it costs ten times. Nothing in the relationship says the loss reaches zero, and nothing says the returns stop being worth it.

You can watch this happen in about ninety seconds, on a source whose E you know exactly. Generate a synthetic language where each symbol depends on the previous four through a fixed random network, so the true conditional entropy is computable: 2.0000 nats per token, against the 2.7726 of guessing uniformly from sixteen symbols. Then train small models on it.

import torch, torch.nn as nn, numpy as np, math, itertools

V, K = 16, 4                                   # 16 symbols, each depends on the last 4
torch.manual_seed(0)
emb = torch.randn(V, 8)
W1 = torch.randn(4 * 8, 96) / math.sqrt(4 * 8)
b1 = torch.randn(96) * 0.1
W2 = torch.randn(96, V) / math.sqrt(96) * 2.5
b2 = torch.randn(V) * 0.1

contexts = torch.tensor(list(itertools.product(range(V), repeat=K)))    # all 65,536 of them
with torch.no_grad():
    P = torch.softmax(torch.tanh(emb[contexts].flatten(1) @ W1 + b1) @ W2 + b2, dim=-1)

cum = P.cumsum(-1).numpy()
N_TOK = 4_500_000
rng = np.random.default_rng(7)
tok = np.zeros(N_TOK, dtype=np.int64)
tok[:K] = rng.integers(0, V, K)
u = rng.random(N_TOK)
idx = 0
for j in range(K):
    idx = idx * V + int(tok[j])
for i in range(K, N_TOK):
    t = min(int(np.searchsorted(cum[idx], u[i])), V - 1)
    tok[i] = t
    idx = (idx * V + t) % (V ** K)

X = torch.tensor(np.stack([tok[j:N_TOK - K + j] for j in range(K)], 1))
Y = torch.tensor(tok[K:])
rows = P[(X * torch.tensor([V ** 3, V ** 2, V, 1])).sum(1)]
H = float(-(rows * rows.clamp_min(1e-12).log()).sum(1).mean())
print("irreducible loss of this source:", round(H, 4))
# irreducible loss of this source: 2.0

Xte, Yte = X[-100_000:], Y[-100_000:]
Xpool, Ypool = X[:-100_000], Y[:-100_000]

class LM(nn.Module):
    def __init__(self, width, dim=8):
        super().__init__()
        self.emb = nn.Embedding(V, dim)
        self.net = nn.Sequential(nn.Linear(K * dim, width), nn.ReLU(), nn.Linear(width, V))
    def forward(self, x):
        return self.net(self.emb(x).flatten(1))

def n_params(width):
    return sum(p.numel() for p in LM(width).net.parameters())   # non-embedding, as papers count it

def train(width, D, seed=0, batch=512, lr=1e-2):
    """One pass over D tokens, cosine decay. Returns held-out loss."""
    torch.manual_seed(seed)
    m = LM(width)
    opt = torch.optim.AdamW(m.parameters(), lr=lr)
    sched = torch.optim.lr_scheduler.CosineAnnealingLR(opt, T_max=max(1, D // batch))
    g = torch.Generator().manual_seed(seed)
    xs, ys = Xpool[:D], Ypool[:D]
    perm = torch.randperm(D, generator=g)
    for s in range(0, D - batch + 1, batch):
        i = perm[s:s + batch]
        opt.zero_grad()
        nn.functional.cross_entropy(m(xs[i]), ys[i]).backward()
        opt.step()
        sched.step()
    m.eval()
    with torch.no_grad():
        return nn.functional.cross_entropy(m(Xte), Yte).item()

def mean3(width, D):
    return float(np.mean([train(width, D, seed=s) for s in range(3)]))

for D in (31_250, 125_000, 500_000, 2_000_000, 4_000_000):
    L = mean3(256, D)
    print(D, round(L, 4), round(L - H, 4))
# 31250    2.1432   0.1432
# 125000   2.0525   0.0525
# 500000   2.0315   0.0315
# 2000000  2.0207   0.0207
# 4000000  2.0172   0.0171

Fitting the excess against tokens over that range gives an exponent of 0.42, and the fit is imperfect: the largest residual is 22% of the excess. Fitting the excess against parameters instead, at a fixed 2,000,000 tokens, gives an exponent of 0.585 and a residual of 91%, which is not a fit at all. The reason is visible once you look at the numbers: past about 800 parameters the model stops being the binding constraint and the data does, so a law in one variable cannot describe it. This is why papers fit a joint form, L = E + A over N to the alpha, plus B over D to the beta, to a grid of runs rather than a line of them.

Compute-optimal allocation#

Given fixed compute, how do you split it between a bigger model and more data? Early large models were substantially undertrained for their size. The compute-optimal finding was that parameters and training tokens should grow together, which is why smaller, longer-trained models began outperforming larger ones.

The method that established it is worth knowing by name and by shape, because it is the one question in this topic an interviewer can ask you to design. It is called an IsoFLOP sweep. Fix a compute budget C. Since training compute is about 6 times parameters times tokens, choosing the model size N fixes the token count at D equal to C divided by 6N. Train a row of models across a wide range of N at that same budget, plot loss against N, and the points make a U: too small a model wastes the data, too large a model gets too few tokens. The bottom of the U is the compute-optimal size for that budget. Repeat at several budgets and you can read off how the optimum moves.

for C in (1.2e9, 4.8e9, 1.92e10, 7.68e10):
    for w in (2, 4, 8, 16, 32, 64, 128, 256, 512):
        N = n_params(w)
        D = int(C / (6 * N))
        if 5_000 <= D <= len(Xpool):
            print(C, N, D, round(D / N, 1), round(mean3(w, D), 4))

C = 4.8e9 FLOPs, so D = C / 6N

parameters N   tokens D    tokens/param   held-out loss
        212     3,773,584       17,800        2.2788   model far too small
        408     1,960,784        4,806        2.1216
        800     1,000,000        1,250        2.0508   bottom of the U
      1,584       505,050          319        2.0526
      3,152       253,807           81        2.0533
      6,288       127,226           20        2.0603
     12,560        63,694            5        2.0836
     25,104        31,867            1        2.1300   model far too large

Fit a parabola to each curve near its minimum, which is what the Chinchilla paper does, and the minima across the four budgets land at 1,220, 1,520, 4,522 and 8,119 parameters. Regressing those against the budgets gives the number this whole literature turns on.

N* proportional to C ^ 0.489
D* proportional to C ^ 0.511

Two exponents near one half is the finding, and it is worth saying plainly what it means. If you get sixteen times the compute, you should make the model four times bigger and train it on four times the data. You should not make it sixteen times bigger. The models that prompted this work had been scaled the other way: a 280-billion-parameter model trained on 300 billion tokens uses 6 times 2.8e11 times 3e11, which is 5.0e23 FLOPs, and a 70-billion-parameter model trained on 1.4 trillion tokens uses 5.9e23, near enough the same budget. The smaller, longer-trained model was the better one, and the difference was not a new architecture but an allocation.

Be careful with the ratio rather than the exponents. Twenty tokens per parameter is the figure quoted from that work, and it is specific to the setup it was measured in. On the toy source above the optimum sits between 130 and 350 tokens per parameter, because a tiny MLP over a four-symbol context is far more data-hungry per parameter than a transformer over text. The transferable result is that the exponents are both near one half; the ratio is something you fit for your own architecture, and it moves with tokenizer, data quality and model shape.

It is also worth knowing that the fitting procedure matters. Fitting the joint form L = E + A over N to the alpha, plus B over D to the beta, to all 27 runs from those sweeps recovers a floor of 1.9884 against a true 2.0000, which is good, but implies N* proportional to C to the 0.36, which disagrees with the 0.489 from the IsoFLOP minima on the same runs. Both are defensible fits to the same data. Papers in this area report several approaches for exactly this reason, and a result that only holds under one fitting procedure is worth less than it looks.

Do they predict? A test#

The point of all this is to predict a run you have not done. So test it rather than assert it. Fit a curve of the form L = E + A times C to the minus b to the minima of the three cheapest sweeps above, extrapolate to the fourth budget, which is four times the largest of them, and then compare with the run.

fitted on the three cheapest:  L(C) = 2.0176 + 947 x C^-0.4603

predicted best loss at C = 7.68e10   2.0269
measured best loss at C = 7.68e10    2.0248

error 0.0020 nats - which is 8% of the 0.0248 the model still has to give

That is the honest character of these predictions. In absolute terms the forecast is excellent and would comfortably justify signing off the larger run. In terms of the quantity a researcher cares about, the remaining headroom, it is out by 8% over one factor of four, and the fitted floor is wrong in a direction that would make you stop scaling too early. Extrapolating three orders of magnitude rather than one is a different proposition, and this is the main reason labs fit a fresh ladder for each architecture rather than reusing published constants.

What they do not tell you#

• Loss is not capability. Predicting loss is not the same as predicting what a model will be able to do.
• They say nothing about which architecture or data mixture. They hold within a setup, not across all of them.
• They are extrapolations, and the relationship could break at scales not yet reached.

Add three more that come up in interviews. They say nothing about post-training: a curve fitted on pretraining loss tells you nothing about what instruction tuning or preference optimisation will add on top, and that gap is where much of the visible quality of a deployed model now comes from. They assume data you do not have: the compute-optimal token count at large budgets exceeds the high-quality text that exists, and repeating data has diminishing and eventually negative returns, so the binding constraint stops being compute. And they are fitted on aggregate held-out loss, which is dominated by the common cases, so a run that improves aggregate loss can be flat or worse on the narrow distribution you care about.

The last one has a practical version worth stating for engineers rather than researchers. If you are choosing between models, a scaling law is not the tool. It predicts the loss of a family of models trained the same way, which is a question about a training run, not about which of two released models does your job better. That is an eval question.

Emergence#

Capabilities are widely reported to appear suddenly at scale. A significant counter-argument holds that this is partly an artefact of discontinuous metrics: a smooth underlying improvement looks like a jump when the measure is all-or-nothing, such as exact-match on a multi-step problem.

That argument is arithmetic and it is worth doing rather than repeating. Take a task that needs ten steps and is scored right or wrong at the end. Suppose scale improves the per-step accuracy smoothly, by five points per doubling of compute, and that the steps are roughly independent. The exact-match score is the per-step accuracy raised to the tenth power.

per-step accuracy   exact match over 10 steps   change
      0.60                    0.0060
      0.65                    0.0135              x2.23
      0.70                    0.0282              x2.10
      0.75                    0.0563              x1.99
      0.80                    0.1074              x1.91
      0.85                    0.1969              x1.83
      0.90                    0.3487              x1.77

Nothing discontinuous happened. Plot the right column against compute and it is a classic emergence figure; plot the left column and it is a straight line. So a claimed emergent ability should always raise the question of what metric produced the curve, and whether a continuous version of the same measurement, such as per-step accuracy, token-level probability of the correct answer, or edit distance to it, also jumps.

The counter-argument to the counter-argument is also real, and a good answer names it. Some measurements are genuinely all-or-nothing in the world, not only on the scoresheet: a model either does or does not write a working exploit, and partial credit is not a thing anybody can deploy. If the discontinuous metric is the one that matters for the decision you are making, then the discontinuity matters for the decision, whatever the underlying curve is doing. The dispute is about what is happening inside the model, not about whether capability arrives usefully in steps.

The dispute is unresolved and worth representing honestly. Confident claims in either direction are ahead of the evidence.

What to skip#

• Skip memorising published constants. The exponents and the tokens-per-parameter ratio belong to the setup they were fitted in, and quoting them as universal is the commonest way to get this wrong in an interview.
• Skip the derivations. There is no derivation; these are fits to runs, and treating them as theory is a category error that shows.
• Worth memorising instead: compute is about 6ND, both compute-optimal exponents are near one half, the fitted form has an irreducible floor, and loss is not capability.