Token

The unit a model actually reads and writes: usually a common word, a word fragment, or a few bytes of text.

What people get wrong

Not a word and not a character. English averages roughly four characters per token, but code, numbers and non-Latin scripts are far less efficient, sometimes a token per character.

A definition is not an understanding. This is taught properly in What a token is, part of Tokenization.

Learn this properly

Terms from the same topic