Entropy, cross-entropy, KL and perplexity

Four numbers turn up constantly in model reporting and are routinely confused with one another. This section separates them, starting from the single quantity all four are averages of, and gives each a plain reading: what the floor is, what your model costs above that floor, what the gap between the two is used for, and what a perplexity figure actually tells you. It also covers when two of these numbers cannot be compared at all, which is the mistake most often made with them in public.

Surprise#

The surprise of an outcome is the negative logarithm of the probability you gave it. Something you said was certain surprises you not at all; something you gave one chance in a thousand surprises you a great deal. Everything on this page is an average of that one quantity, taken over a different set of assumptions.

surprise of an outcome     -log p
entropy        H(p)        average surprise, using the true distribution for both
cross-entropy  H(p, q)     average surprise of your q, when reality is p
KL             KL(p || q)  = H(p, q) - H(p)    the excess you pay for using q

log base 2 gives bits, natural log gives nats. Keep one base throughout.

Entropy is how uncertain the truth itself is: a fair coin has one bit, a loaded one has less, a certain outcome has none. It is the floor. No model, however good, can average less surprise than the entropy of what it is predicting, which is why a training loss that stops falling is not necessarily a model that stopped learning.

Compute it once by hand so the units mean something. A fair coin: each side has probability a half, whose negative log base two is one, and the average of one and one is one bit. A coin that lands heads nine times in ten: heads carries a surprise of 0.152 bits and tails carries 3.322, and the average weighted by how often each happens is 0.9 times 0.152 plus 0.1 times 3.322, which is 0.469 bits. Less than half the fair coin, because you can usually guess. Push it to 99 per cent heads and it falls to 0.081 bits, and at certainty it is zero.

import math

def entropy_bits(p):
    if p in (0.0, 1.0):
        return 0.0
    return -(p * math.log2(p) + (1 - p) * math.log2(1 - p))

for p in (0.5, 0.75, 0.9, 0.99, 1.0):
    print(p, round(entropy_bits(p), 4))
# 0.5 1.0
# 0.75 0.8113
# 0.9 0.469
# 0.99 0.0808
# 1.0 0.0

Cross-entropy and KL#

Cross-entropy is what you actually pay: the average surprise of your predicted distribution against outcomes drawn from the real one. It is the training loss for classification and for next-token prediction. KL divergence is the difference between what you paid and the floor, so it is the part of the loss that is your model's fault rather than the world's.

That split has a direct consequence for how you read a loss number, and it is the reason two training losses are so often compared when they should not be. A loss of 1.6 nats is excellent on text whose own entropy is 1.5 and dreadful on text whose entropy is 0.5, and nothing in the number itself distinguishes the two cases. So a loss curve is only comparable against another run on the same data. A loss that flattens at 1.6 may be a model that has run out of capacity, or a model that has learned everything there is and is now sitting on the floor, and the way to tell them apart is to change the model rather than to stare at the curve.

import math

def kl(p, q):
    return sum(pi * math.log(pi / qi) for pi, qi in zip(p, q))

p = [0.7, 0.2, 0.1]
q = [0.5, 0.3, 0.2]

kl(p, q)     # 0.085 nats
kl(q, p)     # 0.092 nats  -- a different number from the same pair

The asymmetry is not a technicality. KL(p || q) punishes q for putting almost no probability where p has some, so minimising it spreads q out to cover everything. KL(q || p) punishes q for putting probability where p has none, so minimising it makes q concentrate on one mode and ignore the rest. Which direction a method uses decides whether its approximation hedges or commits.

• Preference training uses KL from the starting model as a leash, penalising the tuned model for drifting too far from what it began as.
• Distillation trains the student by minimising KL against the teacher's full output distribution, which carries more information than the label alone.
• Monitoring uses it to detect drift, comparing the distribution of today's inputs against the distribution the model was validated on.

Perplexity#

Perplexity is the exponential of cross-entropy, taken in the same base. It exists because a loss of 1.9 nats per token means nothing to anyone, while its exponential does: a perplexity of 8 says the model is as uncertain, on average, as if it were choosing uniformly between 8 equally likely tokens at each step. That is the average branching factor, and it is the only intuition the number carries.

# Natural-log probabilities the model gave to each correct next token.
logprobs = [-0.10, -2.30, -0.69, -1.61, -0.36]

cross_entropy = -sum(logprobs) / len(logprobs)
perplexity = math.exp(cross_entropy)
print(round(cross_entropy, 3), round(perplexity, 3))     # 1.012 2.751

# Comparable across tokenizers, because bytes are not a modelling choice:
n_bytes = 23
print(round(-sum(logprobs) / math.log(2) / n_bytes, 3))  # 0.317

The branching-factor reading gives entropy a job at inference time as well as in training. Apply the same average-surprise calculation to the model's next-token distribution and its exponential tells you, in effect, how many tokens the model is really choosing between at that step. That number is what a sampling temperature moves. Dividing the logits by a temperature below one sharpens the distribution and lowers its entropy; dividing by a temperature above one flattens it and raises the entropy.

import numpy as np

logits = np.array([4.0, 3.0, 1.0, 0.5, 0.0])
for T in (0.5, 1.0, 2.0):
    z = logits / T
    e = np.exp(z - z.max())
    probs = e / e.sum()
    H = -(probs * np.log(probs)).sum()                 # entropy in nats
    print(T, np.round(probs, 3), round(float(H), 3), round(float(np.exp(H)), 2))

# 0.5 [0.878 0.119 0.002 0.001 0.   ] 0.389 1.48
# 1.0 [0.682 0.251 0.034 0.021 0.012] 0.857 2.36
# 2.0 [0.468 0.284 0.104 0.081 0.063] 1.327 3.77

It is also not a task metric. Perplexity measures how well the model predicts held-out text of the same kind as its training data, which is a different question from whether it does your job. It is a good early warning, a good check on whether quantization damaged something, and a poor argument for shipping.