Using scaling laws: the arithmetic

Scaling laws are usually met as a plot. This section is the arithmetic you can actually do with them. Training compute is about six times parameters times tokens, which is enough to size any run in one line. From there: the compute-optimal split of a budget between model size and data, at roughly twenty tokens per parameter, and then why nearly every production model departs from that ratio on purpose, since it minimises the cost of training and a model is trained once and served forever. It closes on what these laws do not predict, which is when a particular ability turns up.

Compute, roughly#

Training FLOPs  C  ~  6 x N x D
  N = parameters, D = training tokens
  (about 2 per parameter per token forward, 4 backward)

Example: 7B parameters on 2 trillion tokens
  6 x 7e9 x 2e12 = 8.4e22 FLOPs

Where the six comes from is worth a minute, because it is the difference between remembering a formula and being able to adjust it. A forward pass touches each weight once and does a multiply and an add with it, which is 2 FLOPs per parameter per token. The backward pass computes the gradient with respect to the inputs and the gradient with respect to the weights, each costing about as much as the forward pass, so roughly 4 more. Two plus four is six. The estimate leaves out attention over the sequence, which is a real cost that grows with context length and is small relative to the matrix multiplications at the context lengths these models were trained at, and it leaves out the optimiser step, which touches parameters rather than parameters times tokens.

Inference is 2ND rather than 6ND, because there is no backward pass. That factor of three is the whole reason the inference argument later in this section works.

Turn a FLOP count into time with one more division: budget divided by the number of accelerators times what each actually sustains. Take 8.4e22 FLOPs, 1,024 accelerators, and assume each sustains 4e14 FLOP/s on this workload, which is a realistic figure after utilisation rather than a peak number from a datasheet.

8.4e22 / (1024 x 4e14) = 2.05e5 seconds = 2.4 days

Halve the sustained throughput and it is 4.7 days. This is why
utilisation is a first-class engineering concern rather than a detail:
it is a multiplier on a number denominated in weeks and millions.

A compute-optimal split#

The Chinchilla study found that for a fixed compute budget, loss is lowest when parameters and tokens grow together, at roughly 20 training tokens per parameter. Earlier large models had been too big and trained on too little data.

Budget C = 1e21 FLOPs, target D = 20 N
  C = 6 x N x 20N = 120 N^2
  N = sqrt(1e21 / 120) ~ 2.9e9 parameters
  D = 20 x N          ~ 58 billion tokens

The algebra is worth keeping because it answers the question in one line at any budget. Substituting D equal to 20N into 6ND gives 120 N squared, so N is the square root of C over 120 and D is twenty times that. Run it across a few budgets and the pattern is the one the exponents predict: four times the compute buys twice the model and twice the data.

budget C        parameters N     tokens D
  1e21             2.9e9          5.8e10
  1e22             9.1e9          1.8e11
  1e23             2.9e10         5.8e11
  6e23             7.1e10         1.4e12

Why real models ignore that ratio#

Compute-optimal means cheapest to train for a given loss. But a model is trained once and served many times. A smaller model trained on far more data than 20 tokens per parameter costs more to train and much less to serve. Many open models are trained on hundreds or thousands of tokens per parameter for exactly this reason.

Do the arithmetic rather than accepting the argument, because the break-even point is the interesting number. Compare a compute-optimal 70-billion-parameter model trained on 1.4 trillion tokens with a deliberately overtrained 13-billion-parameter model on 8 trillion tokens.

training
  70B on 1.4T tokens   6 x 7.0e10 x 1.4e12 = 5.88e23 FLOPs
  13B on 8.0T tokens   6 x 1.3e10 x 8.0e12 = 6.24e23 FLOPs
  the small model costs 3.6e22 FLOPs more to train: 6% more

inference, per generated token
  70B   2 x 7.0e10 = 1.40e11 FLOPs
  13B   2 x 1.3e10 = 2.60e10 FLOPs
  saving per token                 1.14e11 FLOPs

break-even
  3.6e22 / 1.14e11 = 3.16e11 generated tokens = 316 billion

That is the whole case, and it also tells you when the case fails. A model that will be served rarely, an internal research artefact or a one-off evaluation, should be trained compute-optimally; the overtraining never pays back. The same arithmetic with realistic serving volume is what a system design interviewer is looking for when they ask why anyone would train a small model on trillions of tokens.

There is a second reason, which is the ceiling rather than the cost. Compute-optimal scaling at very large budgets asks for more high-quality tokens than exist, so past some point the data becomes the binding constraint. Repeating data is the usual response and it works for a few epochs before returns fall off, after which the options are better filtering, synthetic data, or accepting a larger model than the ratio wants. The toy sweep in the scaling-laws topic runs into exactly this: at the largest budget there, the compute-optimal model would need more tokens than the corpus has, so the sweep cannot reach it.

Fit a law to your own runs#

Before a large run, train several small models at different sizes, fit a power law of loss against compute, and extrapolate. It is how labs decide whether a large run is worth it, and the same method works at small scale for your own architecture choices.

import numpy as np

compute = np.array([1e17, 3e17, 1e18, 3e18])
loss    = np.array([3.90, 3.62, 3.37, 3.16])       # from your small runs

slope, intercept = np.polyfit(np.log(compute), np.log(loss), 1)
predict = lambda c: np.exp(intercept) * c ** slope
print(f"predicted loss at 1e20: {predict(1e20):.2f}")
# predicted loss at 1e20: 2.54

The straight-line fit above has no floor, so it says the loss keeps falling towards zero for ever. Fit the same four points with a floor and the two disagree more the further you extrapolate.

from scipy.optimize import curve_fit

law = lambda c, E, A, b: E + A * c ** (-b)
(E, A, b), _ = curve_fit(law, compute, loss, p0=[2.0, 50.0, 0.1], maxfev=400000)
print(round(E, 3), round(b, 4))
for c in (1e20, 1e22, 1e30):
    print(f"{c:.0e}  no floor {predict(c):.2f}   with floor {law(c, E, A, b):.2f}")
# 1.836 0.1299
# 1e+20  no floor 2.54   with floor 2.68
# 1e+22  no floor 1.91   with floor 2.30
# 1e+30  no floor 0.62   with floor 1.88

The no-floor fit is optimistic, and it gets more optimistic the further out you go, until it predicts a loss below anything physically achievable. Extrapolating one order of magnitude with either fit is defensible. Extrapolating three is how a plan gets built on a number nobody measured. Fit the floor, quote the range you fitted over, and say how far past it you are projecting.

Loss is not capability#

Scaling laws predict loss on held-out text smoothly. Performance on a specific task can stay flat and then rise quickly, or depend on the metric chosen. Predicting when a particular ability appears is much harder than predicting loss.