Tokenization

Everything you will ever be charged for, and every limit you will ever run into, is counted in tokens. This section explains what one actually is, why the pieces fall where they do, and how to estimate the count of a piece of text before you send it. Code and languages other than English cost noticeably more for the same amount of meaning, which is the sort of thing worth knowing before it turns up on a bill.

Models do not read text#

A language model cannot work with letters. Before anything happens, your text is converted into numbers. Each number stands for a piece of text called a token, and the model only ever sees that list of numbers.

Text:    "Tokenization is fun"

Tokens:  ["Token", "ization", " is", " fun"]

Numbers: [30642, 2065, 374, 2523]

Notice two things. A long word split into two pieces, and the space before a word is part of the token rather than separate. Both are normal.

Why split words at all#

There are two obvious approaches and both fail. One token per character means even a short sentence becomes a very long sequence, and sequences are expensive. One token per word means the vocabulary must contain every word that exists, including every name, typo and new word, which is impossible.

Subword tokens sit in between. Common words get their own token. Rare words are built from pieces. A word the model has never seen can still be represented, because it can always fall back to smaller fragments.

"the"            -> 1 token    very common, gets its own
"tokenization"   -> 2 tokens   less common, built from pieces
"Kowalczyk"      -> 3 tokens   a rare name, built from fragments
"aaaaaaaaaaaa"   -> 4 tokens   nonsense, still representable

Rules of thumb#

For ordinary English prose, one token is roughly four characters, or about three quarters of a word. So 1,000 tokens is about 750 words, and a page of text is around 500 tokens.

Not all text costs the same#

Tokenizers are trained mostly on English, so English is the most efficient thing you can send. Everything else is worse, sometimes dramatically.

• English prose: the baseline, roughly 4 characters per token.
• Code: worse. Indentation, brackets and symbols each tend to become their own token.
• JSON: much worse. Every brace, quote and colon is a token, so structure costs real money.
• Languages not written in Latin script: often two to three times more tokens for the same meaning.
• Long numbers: usually split into several tokens rather than one.

"Hello, how are you?"                    ->  6 tokens
'{"greeting": "Hello", "ok": true}'      -> 13 tokens

You know what a token is; this is how to find out how many you are sending and what that comes to in money. Input and output are priced differently, often by a factor of five, so the same total can cost very different amounts depending on which direction it went in. The section covers measuring before you send, reading the usage figures afterwards, and turning both into a cost per request and per user, which is the number somebody will eventually ask you for.

Count, do not guess#

Every provider exposes the token count of a response, and most let you count text before sending it. Use the real number whenever it affects cost or fits in a limit.

response = client.models.generate_content(
    model="gemini-3.5-flash-lite",
    contents="Explain tokenization in one sentence.",
)

usage = response.usage_metadata
usage.prompt_token_count        # what you sent
usage.candidates_token_count    # what came back

Input and output are priced differently#

Output almost always costs several times more than input. This matters because it changes where you should look when a bill is too high. A long prompt is cheaper than a long answer.

INPUT_PER_MILLION = 1.00
OUTPUT_PER_MILLION = 5.00

def cost(usage) -> float:
    return (
        usage.prompt_token_count / 1_000_000 * INPUT_PER_MILLION
        + usage.candidates_token_count / 1_000_000 * OUTPUT_PER_MILLION
    )

print(f"${cost(usage):.6f}")

Where the cost actually goes#

In a chat or an agent, the input grows every turn because you resend the whole history. By turn ten you are paying to re-read nine previous turns, most of which is the model's own earlier output. That, not the length of any single message, is what makes long conversations expensive.

Turn 1:   7 input tokens
Turn 2:  33 input tokens
Turn 3:  48 input tokens
Turn 10: several hundred, for the same short question

There is a ceiling on how much a model can consider at once, and it is not only your question that counts against it. This section is about working out in advance whether a request will fit, and what to do first when it will not. Exceeding the ceiling is not a graceful degradation: the call fails outright, and it tends to fail on your largest and most important request rather than on a test.

The window#

A model can only consider a fixed number of tokens at once. That budget is the context window, and it covers everything: the system prompt, the conversation history, retrieved documents, tool definitions, tool results, and the answer the model is about to write.

Context window
|<------------------------------------------->|
[ system ][ history ][ documents ][ reply room ]

Exceeding it does not give you a partial answer. The request is rejected. So anything that assembles a prompt from variable-length parts needs to check before sending.

Bigger is not the same as better#

Context windows have grown very large, which tempts people to stop thinking about what they send. Two reasons not to. Cost scales with everything you include, every single call. And recall is not uniform across a long window: material in the middle is retrieved less reliably than material near the beginning or the end.

When it does not fit#

1. 1.Send less. Retrieve the relevant sections instead of the whole document. This is usually both cheaper and more accurate.
2. 2.Summarise older conversation turns and keep the summary instead of the transcript.
3. 3.Split the work into several calls and combine the results.
4. 4.Only then reach for a model with a larger window.

The order matters. Reaching for a bigger window first solves the error and leaves you with a system that is expensive and gets worse as it grows.

Every explanation of tokenization in this topic has treated the tokenizer as given: text goes in, tokens come out, and the mapping looks arbitrary. It is not arbitrary, and the algorithm behind it fits on a page. Writing it is the fastest way to stop being surprised by token counts, and it is the stage the from-scratch GPT project explicitly defers.

The idea#

Start with a vocabulary of single characters, which can represent anything. Count every adjacent pair in the corpus, merge the most frequent pair into a single new symbol, and repeat until the vocabulary is the size you wanted. Frequent sequences — common words, common word endings, common indentation — become single tokens because they were merged early; rare ones stay as several.

The output of training is not a list of tokens. It is an ordered list of merges, and the order is load-bearing: encoding replays the merges in the order they were learned, so a saved tokenizer is that list plus the character inventory it started from.

Splitting before merging#

Real tokenizers do not let merges cross word boundaries, because a token spanning two words would be learned from one corpus and useless in another. Text is first cut into chunks by a regular expression, and merges happen within a chunk. The convention every modern tokenizer follows is to keep a word's leading space attached to it, which is why " the" and "the" are different tokens and why a token count changes when you strip whitespace.

import re
from collections import Counter


def words(text):
    # a word with its leading space, or a run of whitespace on its own
    return re.findall(r" ?[^\s]+|\s+", text)


def merge_pair(symbols, pair):
    a, b = pair
    out, i = [], 0
    while i < len(symbols):
        if i + 1 < len(symbols) and symbols[i] == a and symbols[i + 1] == b:
            out.append(a + b)        # the two symbols become one
            i += 2
        else:
            out.append(symbols[i])
            i += 1
    return tuple(out)

Training#

Count pairs weighted by how often the word containing them appears, rather than walking the raw text, which is the difference between a loop that finishes and one that does not. Ties are broken deterministically, because a tokenizer that trains differently on two runs of the same corpus is a tokenizer you cannot ship.

def train(text, n_merges):
    freqs = Counter(words(text))                 # word -> how many times it occurs
    split = {w: tuple(w) for w in freqs}         # every word starts as its characters
    merges = []

    for _ in range(n_merges):
        pairs = Counter()
        for w, f in freqs.items():
            s = split[w]
            for pair in zip(s, s[1:]):           # every adjacent pair in the word
                pairs[pair] += f                 # weighted by the word's frequency
        if not pairs:
            break

        # most frequent, ties broken on the pair itself so training is reproducible
        best = max(pairs, key=lambda p: (pairs[p], p))
        if pairs[best] < 2:
            break                                # nothing left worth merging

        merges.append(best)
        split = {w: merge_pair(s, best) for w, s in split.items()}

    return merges

corpus = (
    "the cat sat on the mat. the cat sat on the hat. "
    "a cat and a hat and a mat. the rat sat on the cat. "
    "the fat cat sat. the fat rat sat. the mat and the hat."
)

for i, (a, b) in enumerate(train(corpus, 20)):
    print(i, repr(a), "+", repr(b), "->", repr(a + b))

# 0 'a' + 't'    -> 'at'
# 1 't' + 'h'    -> 'th'
# 2 'th' + 'e'   -> 'the'
# 3 ' ' + 'the'  -> ' the'
# 4 'at' + '.'   -> 'at.'
# 5 ' ' + 'a'    -> ' a'
# ...
# 8 ' c' + 'at'  -> ' cat'
# 11 ' s' + 'at' -> ' sat'

Encoding and decoding#

Encoding replays the merge list against a fresh word. Decoding is concatenation and nothing else, which is the property that makes the leading-space convention work: the spaces are inside the tokens, so joining them reproduces the input exactly with no rules about when to insert a space.

def encode(text, merges):
    out = []
    for w in words(text):
        s = tuple(w)
        for pair in merges:          # in the order learned - this is the whole model
            if len(s) == 1:
                break
            s = merge_pair(s, pair)
        out.extend(s)
    return out


def decode(tokens):
    return "".join(tokens)           # the spaces are already in the tokens


merges = train(corpus, 20)
tokens = encode("the fat cat sat on a mat.", merges)
print(tokens)
# ['the', ' ', 'fat', ' cat', ' sat', ' on', ' a', ' mat.']

assert decode(tokens) == "the fat cat sat on a mat."

That split of " fat" into a space and "fat" is the algorithm's whole character in one line. Nothing decided that "cat" deserves its space and "fat" does not; " cat" appeared more often in the training corpus and got its merge before the budget ran out. Every oddity in a real tokenizer has the same shape.

Unseen text, and why bytes#

A word the tokenizer has never seen still encodes: the merges that apply are applied, and whatever is left stays as single characters. Nothing fails.

t = encode("the elephant sat on a submarine.", merges)
print(t)
# ['the', ' ', 'e', 'l', 'e', 'p', 'h', 'a', 'n', 't', ' sat', ' on', ' a',
#  ' s', 'u', 'b', 'm', 'a', 'r', 'i', 'n', 'e', '.']

assert decode(t) == "the elephant sat on a submarine."

That guarantee only holds if the starting inventory covers every possible input, and a character-level start does not: the corpus above has no vocabulary for an emoji or a Cyrillic letter. Real tokenizers start from the 256 possible bytes rather than from characters, so every conceivable input is representable and an unknown token never has to exist. The cost is that one character outside ASCII is several bytes, and so potentially several tokens, before any merges apply.

Turning tokens into ids#

The model needs integers. The vocabulary is the starting inventory followed by the merge results, in merge order, so ids are assigned once at training time and the mapping is part of the saved tokenizer.

vocab = sorted(set(corpus)) + [a + b for a, b in merges]
ids = {t: i for i, t in enumerate(vocab)}

print(len(vocab))                    # 34
print([ids[t] for t in tokens])      # [16, 0, 31, 22, 25, 27, 19, 33]

What this explains#

• Code costs more tokens per character than prose, because indentation, brackets and camel-case identifiers were a smaller share of the training corpus and earned fewer merges. Some tokenizers add explicit merges for runs of spaces, which is why the same code costs different amounts across models.
• Non-English text costs several times more, because byte-level starts mean a character outside ASCII is two to four bytes before merging, and those sequences were rare enough to earn few merges.
• Numbers tokenize unevenly, because "2023" may be one merged token while "2024" is two. This is a documented contributor to arithmetic errors, and it is why several tokenizers now force digits to split into fixed groups.
• A trailing space changes the token count and sometimes the output, because it becomes part of the next word's token rather than a token of its own.
• The same string tokenizes differently under different models, so a token count is only meaningful alongside the tokenizer that produced it.