Asked about Entropy, cross-entropy, KL and perplexity
Why is cross-entropy the thing being minimised rather than entropy?
An answer that works
Entropy is a property of the data and cannot be changed by the model. Cross-entropy is what the model's own distribution costs, so it is the only part that training can move, and the gap between the two is the KL divergence the model is actually reducing.
Reading an answer is not being able to give one. The section that teaches this is Entropy, cross-entropy, KL and perplexity, 4 hours of Probability & statistics.
Learn this properlyAlso asked about Probability & statistics
Your new prompt scores 86 percent against 82 percent on a 50-case eval. Ship it?A test for a condition affecting 1 in 1000 people is 99 percent accurate. You test positive. What is the chance you have it?Why is accuracy misleading on imbalanced data?Why report p95 latency rather than the mean?What is maximum likelihood estimation?Why is cross-entropy the loss for classification?How would you put a confidence interval on an eval score?What does it mean for a model to be calibrated?
All 20 questions on Probability & statistics