Tokenization, section 1 of 4, 2h

What a token is

By the end: You can explain what the model actually reads, and predict roughly how many tokens a piece of text will be.

  • Why models do not read characters or words
  • Subword tokens, and what they look like
  • The rough rules of thumb for English
  • Why code and other languages cost more

New to tokenization? A model does not read letters, and it does not quite read words either. Before any text reaches it, the text is cut into tokens: short pieces of characters drawn from a fixed vocabulary the model was trained on. A common word like "the" is one token; "unhappiness" might arrive as three; a space or a bracket is often a token of its own. Why it matters

Everything you will ever be charged for, and every limit you will ever run into, is counted in tokens. This section explains what one actually is, why the pieces fall where they do, and how to estimate the count of a piece of text before you send it. Code and languages other than English cost noticeably more for the same amount of meaning, which is the sort of thing worth knowing before it turns up on a bill.

Models do not read text#

How text becomes tokens
1 / 5
The quick brown foxFour words to you. Not necessarily four tokens.

Models do not read characters or words. They read tokens.A token is a piece of text the tokenizer learned to treat as one unit.

A language model cannot work with letters. Before anything happens, your text is converted into numbers. Each number stands for a piece of text called a token, and the model only ever sees that list of numbers.

Text:    "Tokenization is fun"

Tokens:  ["Token", "ization", " is", " fun"]

Numbers: [30642, 2065, 374, 2523]
Three words became four tokens. The model sees only the last line.

Notice two things. A long word split into two pieces, and the space before a word is part of the token rather than separate. Both are normal.

Why split words at all#

There are two obvious approaches and both fail. One token per character means even a short sentence becomes a very long sequence, and sequences are expensive. One token per word means the vocabulary must contain every word that exists, including every name, typo and new word, which is impossible.

Subword tokens sit in between. Common words get their own token. Rare words are built from pieces. A word the model has never seen can still be represented, because it can always fall back to smaller fragments.

"the"            -> 1 token    very common, gets its own
"tokenization"   -> 2 tokens   less common, built from pieces
"Kowalczyk"      -> 3 tokens   a rare name, built from fragments
"aaaaaaaaaaaa"   -> 4 tokens   nonsense, still representable

Rules of thumb#

For ordinary English prose, one token is roughly four characters, or about three quarters of a word. So 1,000 tokens is about 750 words, and a page of text is around 500 tokens.

These are estimates, not guarantees

Never build billing or a context-limit check on the four-character rule. Count properly with the tokenizer when the number matters. The estimate is for planning, not for enforcement.

Not all text costs the same#

Tokenizers are trained mostly on English, so English is the most efficient thing you can send. Everything else is worse, sometimes dramatically.

  • English prose: the baseline, roughly 4 characters per token.
  • Code: worse. Indentation, brackets and symbols each tend to become their own token.
  • JSON: much worse. Every brace, quote and colon is a token, so structure costs real money.
  • Languages not written in Latin script: often two to three times more tokens for the same meaning.
  • Long numbers: usually split into several tokens rather than one.
"Hello, how are you?"                    ->  6 tokens
'{"greeting": "Hello", "ok": true}'      -> 13 tokens
The same greeting, wrapped in JSON, costs roughly twice as much.

Where people go wrong

Estimating cost by counting words. The estimate is fine for planning and wrong for anything involving money or a context limit, particularly once your payloads contain JSON or code.

If this comes up in an interview

What is a token?

A piece of text that the model treats as one unit, usually a subword. Text is converted into a list of token ids before the model sees it, and the model only ever works with those ids.

Why use subword tokens rather than words or characters?

Characters make sequences too long, and sequence length drives cost and compute. Whole words need an impossibly large vocabulary and cannot handle unseen words. Subwords give a fixed vocabulary that can still represent anything.

Roughly how many tokens is 1,000 words of English?

About 1,300, using the rule that a token is around three quarters of a word. For anything that matters, count with the tokenizer rather than estimating.

Why does the same content cost more in some languages?

Tokenizers are trained mainly on English text, so English gets efficient single tokens for common words. Other scripts fall back to smaller fragments, often two to three times as many tokens for the same meaning.

Build this: Take the same paragraph in English and in another language, tokenize both, and compare the counts. Then wrap one in JSON and compare again.

Watch it explained

2h 14m

Let's build the GPT Tokenizer

Andrej Karpathy

This is one section of Tokenization, 10 hours across 4 sections. Read it in full to get the whole topic in order.

Next sectionCounting tokens and what they cost

Stop 3 of 50 on the AI Engineer route

0 of 50 topics done · 0h behind you, 603h to go

Next on this routeThe Messages APIStateless request in, one message out.6h

What you have marked done is stored in this browser only. It does not reach a server, and clearing site data will clear it.