Asked about What a token is
What is a token?
An answer that works
A piece of text that the model treats as one unit, usually a subword. Text is converted into a list of token ids before the model sees it, and the model only ever works with those ids.
Reading an answer is not being able to give one. The section that teaches this is What a token is, 2 hours of Tokenization.
Learn this properlyAlso asked about Tokenization
Why use subword tokens rather than words or characters?Roughly how many tokens is 1,000 words of English?Why does the same content cost more in some languages?Why is output usually more expensive than input?Your chatbot costs $0.40 per conversation and needs to be under $0.10. Where do you look first?How would you estimate the monthly cost of a feature before building it?What is a context window?What happens if you exceed it?
All 14 questions on Tokenization