Where capability comes from

Everything a large model can do comes from one task repeated at enormous scale: predict the next token. This section is about why that is enough to produce something that writes code and follows an argument, and about scaling laws, the empirical relationships between compute, data, model size and loss that decide how a training run is planned before a single GPU is booked. It also accounts for a good deal of model behaviour you would otherwise have to memorise: why knowledge stops at a date, why a model is fluent about things it never checked.

One objective, stated precisely#

The objective is worth stating exactly, because most confusion about model behaviour dissolves once you have. At every position in the text the model produces a probability for every token in its vocabulary — a hundred thousand numbers that sum to one. The loss at that position is the negative logarithm of the probability it gave to the token that actually came next. The training loss is that number averaged over every position in the corpus, and it is the only thing the run optimises. Nothing in it mentions truth, helpfulness, reasoning or the user.

The unit is worth knowing too, because it is what people mean when they quote a loss. Measured in nats, a loss of L corresponds to a perplexity of e to the power L, which reads as the number of equally likely tokens the model was effectively choosing between. A model with perplexity 4 is guessing between four options at each step; one with perplexity 1.01 has almost no uncertainty left. Here is the whole objective, at a scale you can read.

import math, collections

text = """the cat sat on the mat . the cat sat on the floor . the dog sat on the mat .
the dog ate the food . the cat ate the food . the cat sat on the chair ."""
tokens = text.split()
vocab = sorted(set(tokens))

# The objective, literally: for every position, a distribution over the vocabulary,
# scored by the log probability it gave to the token that actually came next.
counts = collections.defaultdict(collections.Counter)
for a, b in zip(tokens, tokens[1:]):
    counts[a][b] += 1

def predict(prev):
    c = counts[prev]
    total = sum(c.values())
    return {w: (c[w] + 1) / (total + len(vocab)) for w in vocab}   # add-one smoothing

loss = sum(-math.log(predict(a)[b]) for a, b in zip(tokens, tokens[1:]))
loss /= len(tokens) - 1

print(f"vocabulary {len(vocab)} words, {len(tokens)} tokens")
print("cross-entropy loss  " f"{loss:.3f} nats")
print("perplexity          " f"{math.exp(loss):.1f} words")
print("uniform guessing    " f"{len(vocab):.1f} words")

p = predict("the")
top = sorted(p.items(), key=lambda kv: -kv[1])[:4]
print("after 'the':", ", ".join(f"{w} {q:.2f}" for w, q in top))

# vocabulary 11 words, 40 tokens
# cross-entropy loss  1.513 nats
# perplexity          4.5 words
# uniform guessing    11.0 words
# after 'the': cat 0.22, dog 0.13, food 0.13, mat 0.13

That is the whole of pretraining. Everything else — the transformer, the optimiser, the cluster — is machinery for computing that average over trillions of positions and nudging a few billion numbers to make it smaller.

Why that one task produces general capability#

The usual explanation is that predicting text well requires modelling whatever produced the text, which is true and unsatisfying because it is not measurable. Here is the measurable version. Every regularity in the corpus is a discount on the loss. If some structure in the data — grammar, arithmetic, the fact that a name mentioned earlier tends to reappear — lets you assign more probability to what comes next, then a model that represents that structure scores lower loss than one that does not, and gradient descent will find it. Capability is not a goal of the objective; it is the cheapest way to satisfy it.

You can price one such regularity exactly. Take documents that mention an identifier and then repeat it later. A model with no way to use earlier context can only fall back on how often each identifier appears in the corpus overall. A model that can attend to what it already read just copies.

import math, random, collections

random.seed(0)
ids = [f"{random.randrange(10000):04d}" for _ in range(4000)]
docs = [["opened", i, ";", "wrote", "log", ";", "closed", i] for i in ids]

# Where in each document does the model have to produce the id again?
target = 7

# Model A: no memory of the document. It can only learn the marginal over ids.
marginal = collections.Counter(ids)
n = sum(marginal.values())
loss_a = sum(-math.log(marginal[d[target]] / n) for d in docs) / len(docs)

# Model B: allowed one rule, copy the id it already saw in this document.
loss_b = sum(-math.log(0.999 if d[target] == d[1] else 0.0001) for d in docs) / len(docs)

print(f"{len(docs)} documents, {len(set(ids))} distinct ids")
print("context-blind model   " f"{loss_a:.3f} nats at the id position")
print("copies from context   " f"{loss_b:.3f} nats at the id position")
print("the rule is worth     " f"{loss_a - loss_b:.3f} nats per document")

# 4000 documents, 3308 distinct ids
# context-blind model   8.042 nats at the id position
# copies from context   0.001 nats at the id position
# the rule is worth     8.041 nats per document

Now scale that argument up. Closing a bracket you opened four hundred tokens ago pays. Getting the tense right pays. Knowing that a function defined at the top of the file takes two arguments pays. Knowing that the capital of Peru is Lima pays. So does following an argument well enough to guess its conclusion. Each is a discount, the discounts compound, and none of them were asked for. What you must not conclude is that the model reasons the way you do; the honest claim is only that the cheapest way to predict this corpus turned out to include something that behaves like it much of the time, and behaves like nothing at all the rest.

Scaling laws#

Loss falls predictably as compute, data and parameters increase, following a power law across many orders of magnitude. This is why frontier labs can plan: given a compute budget, you can estimate the loss you will reach and how to split the budget between model size and data.

The version you can use is one line of arithmetic. Training compute is about six FLOPs per parameter per token — two for the forward pass and four for the backward — so a run costs roughly C equals 6 times N times D, where N is parameters and D is training tokens. Everything else is division: by the throughput of a GPU, by the fraction of that throughput you actually achieve, by the hourly price. Do it before you ask anyone for a cluster.

# C = 6 * N * D is the standard approximation: a forward and backward pass
# together cost about six FLOPs per parameter per token. Everything below is
# that one line and a division.
PEAK, MFU, RATE = 989e12, 0.40, 3.00      # H100 bf16 FLOP/s, realised fraction, $/GPU-hour

print("model     params    tokens      FLOPs      GPU-hours         cost   days")
for name, params, per_param, gpus in [
    ("small",  1.5e9,  20,    8),
    ("7B",       7e9,  20,   64),
    ("7B x10",   7e9, 200,   64),
    ("70B",      7e10, 20,  512),
    ("400B",     4e11, 20, 4096),
]:
    tokens = params * per_param
    flops = 6 * params * tokens
    gpu_hours = flops / (PEAK * MFU) / 3600
    days = gpu_hours / gpus / 24
    print(f"{name:8} {params/1e9:6.1f} B {tokens/1e12:7.2f} T  {flops:9.2e} "
          f"{gpu_hours:11,.0f}  " "$" f"{gpu_hours*RATE:11,.0f} {days:6.1f}")
print("(days is wall clock on 8, 64, 64, 512 and 4096 GPUs respectively)")

# model     params    tokens      FLOPs      GPU-hours         cost   days
# small       1.5 B    0.03 T   2.70e+20         190  $        569    1.0
# 7B          7.0 B    0.14 T   5.88e+21       4,129  $     12,386    2.7
# 7B x10      7.0 B    1.40 T   5.88e+22      41,287  $    123,862   26.9
# 70B        70.0 B    1.40 T   5.88e+23     412,875  $  1,238,625   33.6
# 400B      400.0 B    8.00 T   1.92e+25  13,481,631  $ 40,444,894  137.1
# (days is wall clock on 8, 64, 64, 512 and 4096 GPUs respectively)

Three things fall out of that table. A compute-optimal 7B model is a four-thousand-GPU-hour job — about twelve thousand dollars, which is a team's monthly cloud bill rather than a moonshot. Training the same 7B on ten times the data costs ten times as much and is exactly what production models now do, because the extra training is paid once and the smaller model is served forever. And the last row is why the list of organisations doing frontier pretraining is short: forty million dollars of compute, four thousand GPUs held for four and a half months, and that is the successful run, not the failed ones before it.

The other half of scaling is how to split a fixed budget between a bigger model and more data, and C equals 6ND forces the trade: every parameter you add is tokens you can no longer afford. The compute-optimal answer, from the Chinchilla work, is around twenty training tokens per parameter — a ratio, not a number of tokens, so it moves with the budget.

# One budget, five ways to spend it. C = 6ND fixes the trade: every parameter
# you add is tokens you cannot afford.
C = 1e22                                   # FLOPs, roughly two days on 64 H100s

print("params    tokens   tokens/param   verdict")
for params in [1e9, 5e9, 8.3e9, 20e9, 70e9]:
    tokens = C / (6 * params)
    ratio = tokens / params
    if ratio > 60:
        verdict = "small model, drowning in data"
    elif ratio > 8:
        verdict = "near the compute-optimal band"
    else:
        verdict = "big model, badly undertrained"
    print(f"{params/1e9:5.1f} B  {tokens/1e9:7.1f} B  {ratio:11.1f}   {verdict}")

opt = (C / (6 * 20)) ** 0.5                # the size where tokens/param lands on 20
print()
print(f"compute-optimal size at this budget: {opt/1e9:.1f} B parameters")

# params    tokens   tokens/param   verdict
#   1.0 B   1666.7 B       1666.7   small model, drowning in data
#   5.0 B    333.3 B         66.7   small model, drowning in data
#   8.3 B    200.8 B         24.2   near the compute-optimal band
#  20.0 B     83.3 B          4.2   big model, badly undertrained
#  70.0 B     23.8 B          0.3   big model, badly undertrained
#
# compute-optimal size at this budget: 9.1 B parameters

Two caveats keep this honest. Compute-optimal minimises training loss for a training budget, and says nothing about serving cost, which is why almost every model you can actually call is deliberately overtrained — a smaller model pushed well past twenty tokens per parameter is cheaper for every request it ever answers. And at large budgets the recommended token count exceeds the high-quality text that exists, at which point the binding constraint stops being compute and becomes data, which is the subject of the next section.

What this explains#

• Knowledge cutoffs: the model knows the corpus, so it does not know what happened afterwards.
• Confident errors: it was trained to produce plausible continuations, not to signal uncertainty. Fluency and correctness were never separated.
• Uneven competence: it is strong where the corpus was rich and weak where it was thin, which does not track human intuitions about difficulty.
• Format sensitivity: text laid out like its training data is easier to continue well.

Understanding pretraining explains model behaviour better than any amount of prompting folklore, which is the practical reason to learn it even if you will never run one.

Emergence is contested#

What to keep and what to skip#

• Keep: the objective in one sentence, that perplexity is the exponential of the loss, and C equals 6ND. Those three let you answer most of what is asked here.
• Keep: roughly twenty tokens per parameter as the compute-optimal ratio, and that production models are deliberately overtrained past it because inference is paid forever.
• Skip: the exact coefficients of any published scaling law. They are fitted to one lab's setup, they are superseded regularly, and nobody is asking you to recite them.
• Skip: arguing about emergence. Say it is reported, say the metric critique exists, and move on.
• Skip: memorising GPU prices. Memorise the shape of the calculation and look the rate up.