Asked about Byte pair encoding from scratch
Why is a tokenizer trained rather than written?
An answer that works
Because which sequences deserve to be one token is a property of the corpus, not of the language. Frequent sequences get merged early and become single tokens; rare ones stay split. That is why the same tokenizer is efficient on the text it was trained on and expensive on code or on a language that was a small share of the training data, and why the merge list has to be shipped with the model.
Reading an answer is not being able to give one. The section that teaches this is Byte pair encoding from scratch, 4 hours of Tokenization.
Learn this properlyAlso asked about Tokenization
What is a token?Why use subword tokens rather than words or characters?Roughly how many tokens is 1,000 words of English?Why does the same content cost more in some languages?Why is output usually more expensive than input?Your chatbot costs $0.40 per conversation and needs to be under $0.10. Where do you look first?How would you estimate the monthly cost of a feature before building it?What is a context window?
All 14 questions on Tokenization