Asked about What a token is
Why use subword tokens rather than words or characters?
An answer that works
Characters make sequences too long, and sequence length drives cost and compute. Whole words need an impossibly large vocabulary and cannot handle unseen words. Subwords give a fixed vocabulary that can still represent anything.
Reading an answer is not being able to give one. The section that teaches this is What a token is, 2 hours of Tokenization.
Learn this properlyAlso asked about Tokenization
What is a token?Roughly how many tokens is 1,000 words of English?Why does the same content cost more in some languages?Why is output usually more expensive than input?Your chatbot costs $0.40 per conversation and needs to be under $0.10. Where do you look first?How would you estimate the monthly cost of a feature before building it?What is a context window?What happens if you exceed it?
All 14 questions on Tokenization