Asked about Entropy, cross-entropy, KL and perplexity
What is KL divergence?
An answer that works
The average extra surprise from using distribution q when the truth is p: cross-entropy minus entropy. It is zero when the two match and grows as they diverge, and it is measured in bits or nats rather than in any unit of distance.
Then they push further
Why is it not a distance?
It is asymmetric, so KL(p || q) and KL(q || p) are different numbers, and it does not obey the triangle inequality. The direction is a modelling choice: one order makes an approximation spread out to cover everything, the other makes it concentrate on a single mode.
Reading an answer is not being able to give one. The section that teaches this is Entropy, cross-entropy, KL and perplexity, 4 hours of Probability & statistics.
Learn this properlyAlso asked about Probability & statistics