Asked about Entropy, cross-entropy, KL and perplexity
What is perplexity?
An answer that works
The exponential of the average cross-entropy per token. It reads as an average branching factor: a perplexity of 8 means the model is as uncertain as if it were choosing uniformly among 8 tokens at each step. Lower means less surprised by the text.
Then they push further
One model reports 12 and another reports 9. Is the second better?
Only if they share a tokenizer and were measured on the same text. Perplexity is per token, so a vocabulary that splits text into fewer, longer tokens carries more information per token and scores worse for no reason connected to quality.
How would you compare two models with different tokenizers?
Convert to bits per byte or bits per character: total negative log probability over the text, divided by the number of bytes rather than the number of tokens. Bytes are a property of the text, not a modelling choice.
Perplexity improved and your eval got worse. What happened?
They measure different things. Perplexity is fit to held-out text of a similar kind, and a model can predict such text better while doing your task worse, particularly after training on data that resembles the perplexity set more than it resembles production traffic. The task eval is the one that decides.
What gets marked up
Strong answers give the exponential-of-cross-entropy definition, the branching factor reading, and the tokenizer caveat without being prompted.
Reading an answer is not being able to give one. The section that teaches this is Entropy, cross-entropy, KL and perplexity, 4 hours of Probability & statistics.
Learn this properlyAlso asked about Probability & statistics