Scaling laws

Loss falls predictably as you add compute, data and parameters, and it does so across many orders of magnitude. That regularity is what justifies spending tens of millions on a training run before anyone knows what the model will be able to do. This section covers the relationship, how it settles the trade-off between model size and data volume for a fixed budget, and its limits — it predicts loss, not capability, which is where the argument over emergent abilities sits.

The relationship#

Loss falls as a power law in compute, data and parameters, holding across many orders of magnitude. It is an empirical regularity rather than a theorem, and it is remarkably consistent.

Say what a power law actually claims, because the phrase is often used to mean no more than 'it goes down'. The form fitted is a floor plus a decaying term. Written for parameters, it is L(N) = E + A times N to the power of minus alpha, where L is the loss, N is the number of parameters, E is the loss you would reach with an infinitely large model, and alpha sets how fast you approach it. The important word is E. Text has an irreducible unpredictability, and no amount of compute removes it, so what falls as a power law is the excess over E rather than the loss itself. A paper reporting a smooth straight line on a log-log plot has usually subtracted an E it also fitted.

The consequence for reading such a plot is that a fixed factor of improvement gets more expensive each time. If alpha is 0.5, halving the excess costs four times the parameters. If alpha is 0.3, it costs ten times. Nothing in the relationship says the loss reaches zero, and nothing says the returns stop being worth it.

You can watch this happen in about ninety seconds, on a source whose E you know exactly. Generate a synthetic language where each symbol depends on the previous four through a fixed random network, so the true conditional entropy is computable: 2.0000 nats per token, against the 2.7726 of guessing uniformly from sixteen symbols. Then train small models on it.

import torch, torch.nn as nn, numpy as np, math, itertools

V, K = 16, 4                                   # 16 symbols, each depends on the last 4
torch.manual_seed(0)
emb = torch.randn(V, 8)
W1 = torch.randn(4 * 8, 96) / math.sqrt(4 * 8)
b1 = torch.randn(96) * 0.1
W2 = torch.randn(96, V) / math.sqrt(96) * 2.5
b2 = torch.randn(V) * 0.1

contexts = torch.tensor(list(itertools.product(range(V), repeat=K)))    # all 65,536 of them
with torch.no_grad():
    P = torch.softmax(torch.tanh(emb[contexts].flatten(1) @ W1 + b1) @ W2 + b2, dim=-1)

cum = P.cumsum(-1).numpy()
N_TOK = 4_500_000
rng = np.random.default_rng(7)
tok = np.zeros(N_TOK, dtype=np.int64)
tok[:K] = rng.integers(0, V, K)
u = rng.random(N_TOK)
idx = 0
for j in range(K):
    idx = idx * V + int(tok[j])
for i in range(K, N_TOK):
    t = min(int(np.searchsorted(cum[idx], u[i])), V - 1)
    tok[i] = t
    idx = (idx * V + t) % (V ** K)

X = torch.tensor(np.stack([tok[j:N_TOK - K + j] for j in range(K)], 1))
Y = torch.tensor(tok[K:])
rows = P[(X * torch.tensor([V ** 3, V ** 2, V, 1])).sum(1)]
H = float(-(rows * rows.clamp_min(1e-12).log()).sum(1).mean())
print("irreducible loss of this source:", round(H, 4))
# irreducible loss of this source: 2.0

Xte, Yte = X[-100_000:], Y[-100_000:]
Xpool, Ypool = X[:-100_000], Y[:-100_000]

class LM(nn.Module):
    def __init__(self, width, dim=8):
        super().__init__()
        self.emb = nn.Embedding(V, dim)
        self.net = nn.Sequential(nn.Linear(K * dim, width), nn.ReLU(), nn.Linear(width, V))
    def forward(self, x):
        return self.net(self.emb(x).flatten(1))

def n_params(width):
    return sum(p.numel() for p in LM(width).net.parameters())   # non-embedding, as papers count it

def train(width, D, seed=0, batch=512, lr=1e-2):
    """One pass over D tokens, cosine decay. Returns held-out loss."""
    torch.manual_seed(seed)
    m = LM(width)
    opt = torch.optim.AdamW(m.parameters(), lr=lr)
    sched = torch.optim.lr_scheduler.CosineAnnealingLR(opt, T_max=max(1, D // batch))
    g = torch.Generator().manual_seed(seed)
    xs, ys = Xpool[:D], Ypool[:D]
    perm = torch.randperm(D, generator=g)
    for s in range(0, D - batch + 1, batch):
        i = perm[s:s + batch]
        opt.zero_grad()
        nn.functional.cross_entropy(m(xs[i]), ys[i]).backward()
        opt.step()
        sched.step()
    m.eval()
    with torch.no_grad():
        return nn.functional.cross_entropy(m(Xte), Yte).item()

def mean3(width, D):
    return float(np.mean([train(width, D, seed=s) for s in range(3)]))

for D in (31_250, 125_000, 500_000, 2_000_000, 4_000_000):
    L = mean3(256, D)
    print(D, round(L, 4), round(L - H, 4))
# 31250    2.1432   0.1432
# 125000   2.0525   0.0525
# 500000   2.0315   0.0315
# 2000000  2.0207   0.0207
# 4000000  2.0172   0.0171

Fitting the excess against tokens over that range gives an exponent of 0.42, and the fit is imperfect: the largest residual is 22% of the excess. Fitting the excess against parameters instead, at a fixed 2,000,000 tokens, gives an exponent of 0.585 and a residual of 91%, which is not a fit at all. The reason is visible once you look at the numbers: past about 800 parameters the model stops being the binding constraint and the data does, so a law in one variable cannot describe it. This is why papers fit a joint form, L = E + A over N to the alpha, plus B over D to the beta, to a grid of runs rather than a line of them.

Compute-optimal allocation#

Given fixed compute, how do you split it between a bigger model and more data? Early large models were substantially undertrained for their size. The compute-optimal finding was that parameters and training tokens should grow together, which is why smaller, longer-trained models began outperforming larger ones.

The method that established it is worth knowing by name and by shape, because it is the one question in this topic an interviewer can ask you to design. It is called an IsoFLOP sweep. Fix a compute budget C. Since training compute is about 6 times parameters times tokens, choosing the model size N fixes the token count at D equal to C divided by 6N. Train a row of models across a wide range of N at that same budget, plot loss against N, and the points make a U: too small a model wastes the data, too large a model gets too few tokens. The bottom of the U is the compute-optimal size for that budget. Repeat at several budgets and you can read off how the optimum moves.

for C in (1.2e9, 4.8e9, 1.92e10, 7.68e10):
    for w in (2, 4, 8, 16, 32, 64, 128, 256, 512):
        N = n_params(w)
        D = int(C / (6 * N))
        if 5_000 <= D <= len(Xpool):
            print(C, N, D, round(D / N, 1), round(mean3(w, D), 4))

C = 4.8e9 FLOPs, so D = C / 6N

parameters N   tokens D    tokens/param   held-out loss
        212     3,773,584       17,800        2.2788   model far too small
        408     1,960,784        4,806        2.1216
        800     1,000,000        1,250        2.0508   bottom of the U
      1,584       505,050          319        2.0526
      3,152       253,807           81        2.0533
      6,288       127,226           20        2.0603
     12,560        63,694            5        2.0836
     25,104        31,867            1        2.1300   model far too large

Fit a parabola to each curve near its minimum, which is what the Chinchilla paper does, and the minima across the four budgets land at 1,220, 1,520, 4,522 and 8,119 parameters. Regressing those against the budgets gives the number this whole literature turns on.

N* proportional to C ^ 0.489
D* proportional to C ^ 0.511

Two exponents near one half is the finding, and it is worth saying plainly what it means. If you get sixteen times the compute, you should make the model four times bigger and train it on four times the data. You should not make it sixteen times bigger. The models that prompted this work had been scaled the other way: a 280-billion-parameter model trained on 300 billion tokens uses 6 times 2.8e11 times 3e11, which is 5.0e23 FLOPs, and a 70-billion-parameter model trained on 1.4 trillion tokens uses 5.9e23, near enough the same budget. The smaller, longer-trained model was the better one, and the difference was not a new architecture but an allocation.

Be careful with the ratio rather than the exponents. Twenty tokens per parameter is the figure quoted from that work, and it is specific to the setup it was measured in. On the toy source above the optimum sits between 130 and 350 tokens per parameter, because a tiny MLP over a four-symbol context is far more data-hungry per parameter than a transformer over text. The transferable result is that the exponents are both near one half; the ratio is something you fit for your own architecture, and it moves with tokenizer, data quality and model shape.

It is also worth knowing that the fitting procedure matters. Fitting the joint form L = E + A over N to the alpha, plus B over D to the beta, to all 27 runs from those sweeps recovers a floor of 1.9884 against a true 2.0000, which is good, but implies N* proportional to C to the 0.36, which disagrees with the 0.489 from the IsoFLOP minima on the same runs. Both are defensible fits to the same data. Papers in this area report several approaches for exactly this reason, and a result that only holds under one fitting procedure is worth less than it looks.

Do they predict? A test#

The point of all this is to predict a run you have not done. So test it rather than assert it. Fit a curve of the form L = E + A times C to the minus b to the minima of the three cheapest sweeps above, extrapolate to the fourth budget, which is four times the largest of them, and then compare with the run.

fitted on the three cheapest:  L(C) = 2.0176 + 947 x C^-0.4603

predicted best loss at C = 7.68e10   2.0269
measured best loss at C = 7.68e10    2.0248

error 0.0020 nats - which is 8% of the 0.0248 the model still has to give

That is the honest character of these predictions. In absolute terms the forecast is excellent and would comfortably justify signing off the larger run. In terms of the quantity a researcher cares about, the remaining headroom, it is out by 8% over one factor of four, and the fitted floor is wrong in a direction that would make you stop scaling too early. Extrapolating three orders of magnitude rather than one is a different proposition, and this is the main reason labs fit a fresh ladder for each architecture rather than reusing published constants.

What they do not tell you#

• Loss is not capability. Predicting loss is not the same as predicting what a model will be able to do.
• They say nothing about which architecture or data mixture. They hold within a setup, not across all of them.
• They are extrapolations, and the relationship could break at scales not yet reached.

Add three more that come up in interviews. They say nothing about post-training: a curve fitted on pretraining loss tells you nothing about what instruction tuning or preference optimisation will add on top, and that gap is where much of the visible quality of a deployed model now comes from. They assume data you do not have: the compute-optimal token count at large budgets exceeds the high-quality text that exists, and repeating data has diminishing and eventually negative returns, so the binding constraint stops being compute. And they are fitted on aggregate held-out loss, which is dominated by the common cases, so a run that improves aggregate loss can be flat or worse on the narrow distribution you care about.

The last one has a practical version worth stating for engineers rather than researchers. If you are choosing between models, a scaling law is not the tool. It predicts the loss of a family of models trained the same way, which is a question about a training run, not about which of two released models does your job better. That is an eval question.

Emergence#

Capabilities are widely reported to appear suddenly at scale. A significant counter-argument holds that this is partly an artefact of discontinuous metrics: a smooth underlying improvement looks like a jump when the measure is all-or-nothing, such as exact-match on a multi-step problem.

That argument is arithmetic and it is worth doing rather than repeating. Take a task that needs ten steps and is scored right or wrong at the end. Suppose scale improves the per-step accuracy smoothly, by five points per doubling of compute, and that the steps are roughly independent. The exact-match score is the per-step accuracy raised to the tenth power.

per-step accuracy   exact match over 10 steps   change
      0.60                    0.0060
      0.65                    0.0135              x2.23
      0.70                    0.0282              x2.10
      0.75                    0.0563              x1.99
      0.80                    0.1074              x1.91
      0.85                    0.1969              x1.83
      0.90                    0.3487              x1.77

Nothing discontinuous happened. Plot the right column against compute and it is a classic emergence figure; plot the left column and it is a straight line. So a claimed emergent ability should always raise the question of what metric produced the curve, and whether a continuous version of the same measurement, such as per-step accuracy, token-level probability of the correct answer, or edit distance to it, also jumps.

The counter-argument to the counter-argument is also real, and a good answer names it. Some measurements are genuinely all-or-nothing in the world, not only on the scoresheet: a model either does or does not write a working exploit, and partial credit is not a thing anybody can deploy. If the discontinuous metric is the one that matters for the decision you are making, then the discontinuity matters for the decision, whatever the underlying curve is doing. The dispute is about what is happening inside the model, not about whether capability arrives usefully in steps.

The dispute is unresolved and worth representing honestly. Confident claims in either direction are ahead of the evidence.

What to skip#

• Skip memorising published constants. The exponents and the tokens-per-parameter ratio belong to the setup they were fitted in, and quoting them as universal is the commonest way to get this wrong in an interview.
• Skip the derivations. There is no derivation; these are fits to runs, and treating them as theory is a category error that shows.
• Worth memorising instead: compute is about 6ND, both compute-optimal exponents are near one half, the fitted form has an irreducible floor, and loss is not capability.

Scaling laws are usually met as a plot. This section is the arithmetic you can actually do with them. Training compute is about six times parameters times tokens, which is enough to size any run in one line. From there: the compute-optimal split of a budget between model size and data, at roughly twenty tokens per parameter, and then why nearly every production model departs from that ratio on purpose, since it minimises the cost of training and a model is trained once and served forever. It closes on what these laws do not predict, which is when a particular ability turns up.

Compute, roughly#

Training FLOPs  C  ~  6 x N x D
  N = parameters, D = training tokens
  (about 2 per parameter per token forward, 4 backward)

Example: 7B parameters on 2 trillion tokens
  6 x 7e9 x 2e12 = 8.4e22 FLOPs

Where the six comes from is worth a minute, because it is the difference between remembering a formula and being able to adjust it. A forward pass touches each weight once and does a multiply and an add with it, which is 2 FLOPs per parameter per token. The backward pass computes the gradient with respect to the inputs and the gradient with respect to the weights, each costing about as much as the forward pass, so roughly 4 more. Two plus four is six. The estimate leaves out attention over the sequence, which is a real cost that grows with context length and is small relative to the matrix multiplications at the context lengths these models were trained at, and it leaves out the optimiser step, which touches parameters rather than parameters times tokens.

Inference is 2ND rather than 6ND, because there is no backward pass. That factor of three is the whole reason the inference argument later in this section works.

Turn a FLOP count into time with one more division: budget divided by the number of accelerators times what each actually sustains. Take 8.4e22 FLOPs, 1,024 accelerators, and assume each sustains 4e14 FLOP/s on this workload, which is a realistic figure after utilisation rather than a peak number from a datasheet.

8.4e22 / (1024 x 4e14) = 2.05e5 seconds = 2.4 days

Halve the sustained throughput and it is 4.7 days. This is why
utilisation is a first-class engineering concern rather than a detail:
it is a multiplier on a number denominated in weeks and millions.

A compute-optimal split#

The Chinchilla study found that for a fixed compute budget, loss is lowest when parameters and tokens grow together, at roughly 20 training tokens per parameter. Earlier large models had been too big and trained on too little data.

Budget C = 1e21 FLOPs, target D = 20 N
  C = 6 x N x 20N = 120 N^2
  N = sqrt(1e21 / 120) ~ 2.9e9 parameters
  D = 20 x N          ~ 58 billion tokens

The algebra is worth keeping because it answers the question in one line at any budget. Substituting D equal to 20N into 6ND gives 120 N squared, so N is the square root of C over 120 and D is twenty times that. Run it across a few budgets and the pattern is the one the exponents predict: four times the compute buys twice the model and twice the data.

budget C        parameters N     tokens D
  1e21             2.9e9          5.8e10
  1e22             9.1e9          1.8e11
  1e23             2.9e10         5.8e11
  6e23             7.1e10         1.4e12

Why real models ignore that ratio#

Compute-optimal means cheapest to train for a given loss. But a model is trained once and served many times. A smaller model trained on far more data than 20 tokens per parameter costs more to train and much less to serve. Many open models are trained on hundreds or thousands of tokens per parameter for exactly this reason.

Do the arithmetic rather than accepting the argument, because the break-even point is the interesting number. Compare a compute-optimal 70-billion-parameter model trained on 1.4 trillion tokens with a deliberately overtrained 13-billion-parameter model on 8 trillion tokens.

training
  70B on 1.4T tokens   6 x 7.0e10 x 1.4e12 = 5.88e23 FLOPs
  13B on 8.0T tokens   6 x 1.3e10 x 8.0e12 = 6.24e23 FLOPs
  the small model costs 3.6e22 FLOPs more to train: 6% more

inference, per generated token
  70B   2 x 7.0e10 = 1.40e11 FLOPs
  13B   2 x 1.3e10 = 2.60e10 FLOPs
  saving per token                 1.14e11 FLOPs

break-even
  3.6e22 / 1.14e11 = 3.16e11 generated tokens = 316 billion

That is the whole case, and it also tells you when the case fails. A model that will be served rarely, an internal research artefact or a one-off evaluation, should be trained compute-optimally; the overtraining never pays back. The same arithmetic with realistic serving volume is what a system design interviewer is looking for when they ask why anyone would train a small model on trillions of tokens.

There is a second reason, which is the ceiling rather than the cost. Compute-optimal scaling at very large budgets asks for more high-quality tokens than exist, so past some point the data becomes the binding constraint. Repeating data is the usual response and it works for a few epochs before returns fall off, after which the options are better filtering, synthetic data, or accepting a larger model than the ratio wants. The toy sweep in the scaling-laws topic runs into exactly this: at the largest budget there, the compute-optimal model would need more tokens than the corpus has, so the sweep cannot reach it.

Fit a law to your own runs#

Before a large run, train several small models at different sizes, fit a power law of loss against compute, and extrapolate. It is how labs decide whether a large run is worth it, and the same method works at small scale for your own architecture choices.

import numpy as np

compute = np.array([1e17, 3e17, 1e18, 3e18])
loss    = np.array([3.90, 3.62, 3.37, 3.16])       # from your small runs

slope, intercept = np.polyfit(np.log(compute), np.log(loss), 1)
predict = lambda c: np.exp(intercept) * c ** slope
print(f"predicted loss at 1e20: {predict(1e20):.2f}")
# predicted loss at 1e20: 2.54

The straight-line fit above has no floor, so it says the loss keeps falling towards zero for ever. Fit the same four points with a floor and the two disagree more the further you extrapolate.

from scipy.optimize import curve_fit

law = lambda c, E, A, b: E + A * c ** (-b)
(E, A, b), _ = curve_fit(law, compute, loss, p0=[2.0, 50.0, 0.1], maxfev=400000)
print(round(E, 3), round(b, 4))
for c in (1e20, 1e22, 1e30):
    print(f"{c:.0e}  no floor {predict(c):.2f}   with floor {law(c, E, A, b):.2f}")
# 1.836 0.1299
# 1e+20  no floor 2.54   with floor 2.68
# 1e+22  no floor 1.91   with floor 2.30
# 1e+30  no floor 0.62   with floor 1.88

The no-floor fit is optimistic, and it gets more optimistic the further out you go, until it predicts a loss below anything physically achievable. Extrapolating one order of magnitude with either fit is defensible. Extrapolating three is how a plan gets built on a number nobody measured. Fit the floor, quote the range you fitted over, and say how far past it you are projecting.

Loss is not capability#

Scaling laws predict loss on held-out text smoothly. Performance on a specific task can stay flat and then rise quickly, or depend on the metric chosen. Predicting when a particular ability appears is much harder than predicting loss.