Sampling & decoding interview questions
8 questions across 3 sections, in the order you would learn them. Every answer here is one that works — but an answer you have read is not an answer you can give, so use Test yourself and say it out loud first.
How a model picks the next token
Read the section that teaches thisIf this comes up in an interview
What is the difference between temperature and top-p?
Temperature rescales the whole probability distribution, making peaks sharper or flatter. top-p truncates it, keeping the smallest set of tokens whose probabilities sum to p and sampling among those. Change one of them, not both, or the interaction becomes hard to reason about.
Is temperature 0 deterministic?
Not on a hosted API. Batched GPU execution, non-associative floating point addition, expert routing and varying hardware all introduce small differences. Locally, with a fixed seed and a single request, it can be.
You need consistent JSON output. What settings do you use?
Low temperature, around 0 to 0.2, and schema-constrained decoding if available, which makes structure a guarantee rather than a preference. Then check the stop reason, because truncation produces invalid JSON regardless of temperature.
Sampling settings for real tasks
Read the section that teaches thisIf this comes up in an interview
What is self-consistency?
Sampling several answers to the same question and taking the most common one. Independent errors rarely agree, so the majority answer is more accurate than one sample. It costs one call per sample, so it suits high value questions with a single correct answer.
Then they push further
What does it mean when the samples disagree?
That the model is uncertain on this input. That is useful: route those cases to a person or a stronger model instead of trusting the majority of a split vote.
Why does it not help for open-ended writing?
There is no single answer to vote on. For writing you would need a judge or a person to choose between samples, which is a different technique with its own costs.
What gets marked up. Marked up for naming the cost and for using disagreement between samples as a confidence signal.
Your answers are sometimes cut off mid-sentence. Why?
The response hit max_tokens. The model does not plan around that limit, it is simply stopped. Check the stop reason, raise the limit or ask for shorter output in the prompt.
Sampling from scratch in NumPy
Read the section that teaches thisIf this comes up in an interview
Implement top-p sampling given a vector of logits.
Divide the logits by the temperature, sort descending, softmax, take the cumulative sum, and find the first index where it reaches p. Keep everything up to and including that index, renormalise those probabilities so they sum to one, and draw from them. The inclusive cut matters: without it a step where the top token alone exceeds p keeps an empty set.
Then they push further
Why not just use a fixed k?
Because the right number of candidates depends on how confident the model is at that step, and that varies enormously. On a step where one token holds 95% of the mass, a nucleus at 0.9 is one token and k=10 admits nine the model had ruled out. On a genuinely open step the nucleus might be forty tokens and k=10 cuts off thirty the model thought equally plausible. Top-p reads the cut off the distribution instead of fixing it in advance.
Does temperature change which tokens survive top-k?
No, because dividing by a positive number is monotonic, so the ranking is unchanged and the same k tokens survive. It does change the probabilities they are then sampled with. It changes a top-p set, though: flattening the distribution spreads the mass, so more tokens are needed to reach the threshold.
What gets marked up. The inclusive cut and the renormalisation are what get checked. Sorting and cumulative-summing is the easy part; candidates who return an empty set on a confident step, or who forget to renormalise after truncating, are sampling from something that is not a distribution.
Why is greedy decoding the same as temperature zero?
Dividing the logits by a temperature approaching zero sends every gap between logits to infinity, so the softmax converges to one on the largest logit and zero everywhere else. Sampling from that distribution always returns the argmax. In code you special-case it rather than dividing by zero, but it is a limit and not an exception.
Why use argpartition rather than argsort for top-k?
Top-k needs the identity of the k largest, not their order, and argpartition finds them in time linear in the vocabulary rather than the n log n a full sort costs. On a vocabulary of a hundred thousand, executed for every token generated, that is a real difference. Nucleus sampling does need the sort, because it depends on the running total in descending order.
These are the questions for one topic. The whole question bank covers every topic on the site.