Tokenization interview questions

What is a token?

A piece of text that the model treats as one unit, usually a subword. Text is converted into a list of token ids before the model sees it, and the model only ever works with those ids.

Why use subword tokens rather than words or characters?

Characters make sequences too long, and sequence length drives cost and compute. Whole words need an impossibly large vocabulary and cannot handle unseen words. Subwords give a fixed vocabulary that can still represent anything.

Roughly how many tokens is 1,000 words of English?

About 1,300, using the rule that a token is around three quarters of a word. For anything that matters, count with the tokenizer rather than estimating.

Why does the same content cost more in some languages?

Tokenizers are trained mainly on English text, so English gets efficient single tokens for common words. Other scripts fall back to smaller fragments, often two to three times as many tokens for the same meaning.

Why is output usually more expensive than input?

Generating tokens is sequential work: the model produces one token at a time, each conditioned on all the previous ones. Reading input is done in one parallel pass. The pricing reflects the difference in compute.

Your chatbot costs $0.40 per conversation and needs to be under $0.10. Where do you look first?

Input tokens across the whole conversation, because history is resent every turn and grows quadratically. Prompt caching on the stable prefix, trimming or summarising old turns, and routing simple turns to a cheaper model, in that order.

How would you estimate the monthly cost of a feature before building it?

Take a representative prompt and response, count both, apply the per-million rates, and multiply by expected requests. Add a margin for retries and for conversations being longer than the sample.

What is a context window?

The maximum number of tokens a model can process in one request. It includes the system prompt, the full conversation history, any retrieved content, tool definitions and the response being generated.

What happens if you exceed it?

The request is rejected rather than truncated. Anything building a prompt from variable-length pieces has to check the size before sending.

If a model has a very large context window, why not put everything in it?

Cost scales with every token on every call, latency grows, and recall across a long window is uneven, with material in the middle attended to least reliably. Retrieving the relevant part is usually both cheaper and more accurate.

A user uploads a 500-page PDF and asks a question about it. How do you handle it?

Do not send all of it. Chunk the document, index it, retrieve the sections relevant to the question, and send only those with citations back to the source. That keeps cost flat as documents grow and lets you show where the answer came from.

What gets marked up. Strong answers separate questions that need one part of the document from ones that need all of it.

Implement byte pair encoding: training, encoding and decoding.

Training: split the corpus into words, represent each as a tuple of bytes or characters, then repeatedly count every adjacent pair weighted by word frequency, merge the most frequent pair everywhere, and record it. The output is the ordered merge list. Encoding: split the input the same way and replay the merges in order against each word. Decoding: concatenate the token strings, which works exactly because leading spaces are kept inside the tokens.

What gets marked up. Candidates who have written it keep the leading space inside the token and know decoding is plain concatenation. Handling a space as its own token, and then needing rules about when to re-insert it, is the sign of someone recalling the idea rather than having implemented it.

Why is a tokenizer trained rather than written?

Because which sequences deserve to be one token is a property of the corpus, not of the language. Frequent sequences get merged early and become single tokens; rare ones stay split. That is why the same tokenizer is efficient on the text it was trained on and expensive on code or on a language that was a small share of the training data, and why the merge list has to be shipped with the model.

Why do models struggle with arithmetic and with spelling questions?

Both follow from the segmentation. A number may be one token or three depending on whether that digit sequence was frequent enough to merge, so digits that should be treated positionally arrive in inconsistent groupings — several tokenizers now force digits into fixed groups for exactly this reason. Spelling questions fail for the same reason from the other direction: a word that arrives as a single token gives the model no direct view of its letters.