How a model picks the next token
By the end: You can choose sampling settings on purpose and explain what each one does.
- The probability distribution over the vocabulary
- Temperature, top-p and top-k
- Which to change and which to leave alone
- Why temperature zero is not reproducible
New to sampling & decoding? A model never picks the next token outright. It produces a probability for every token it could say next, and something has to choose one from that spread. Sampling is that choice, and temperature and top-p are the dials on it: low settings make the model take the likeliest option almost every time, higher settings let less likely ones through, which reads as more variety and more invention. Why it matters
A model does not decide on an answer. At each step it produces a probability for every token it could write next, and something has to pick one of them. That picking is sampling, and the settings you have seen copied into every code example, temperature, top-p and top-k, are all ways of reshaping those probabilities before the draw. This section is about what each one actually does, and why even the most deterministic setting on offer will not guarantee the same output twice.
Every step is a choice#
At each step the model produces a probability for every token in its vocabulary. Sampling is how one of them gets picked. All the settings you can change are ways of reshaping that distribution before the draw.
After "The capital of France is":
" Paris" 0.94
" located" 0.02
" a" 0.01
...everything else shares the remaining 0.03Temperature#
Temperature divides the raw scores, the logits, before they are turned into probabilities. Below 1 the peaks get sharper and the likely token becomes almost certain. Above 1 the distribution flattens and unlikely tokens get a real chance.
Original: Paris 0.94 located 0.02 a 0.01
t = 0.2: Paris 0.99+ everything else vanishes
t = 1.0: unchanged
t = 1.8: Paris 0.60 others rise substantiallyIt cannot invent uncertainty
If the model is confident, no temperature setting will change the answer much. Asking for one colour gives blue at temperature 0 and at temperature 2. High temperature only creates variety where the model was already unsure, which is why it is not a creativity dial.
top-p and top-k#
These truncate rather than rescale. top-k keeps the k most likely tokens and discards the rest. top-p, also called nucleus sampling, keeps the smallest set of tokens whose probabilities add up to p.
top_p = 0.9 keep tokens until their probabilities sum to 0.9,
then sample among those
Adaptive: when the model is confident, that is one or two tokens.
When it is unsure, it is many.In practice, change one of them. Tuning temperature and top-p together makes the effect of each hard to reason about. Most people leave top-p at its default and adjust temperature.
Choosing#
- Extraction, classification, anything parsed, anything evaluated: 0 to 0.3.
- General assistant output: around 0.7, which is usually the default.
- Creative writing and brainstorming: 0.8 to 1.0.
- Above roughly 1.5: degrades quickly and rarely useful.
Zero is not deterministic#
Temperature 0 means always take the most likely token — greedy decoding — which sounds reproducible. On a hosted API it is not. Requests are batched onto GPUs, floating point addition is not exactly associative so the order of a batched reduction changes the last bits, mixture-of-experts routing can depend on batch composition, and different requests land on different hardware.
"Name any animal", temperature 0, five runs:
Elephant, Lion, Lion, Lion, Elephant
Same prompt, same setting, different answers.Where people go wrong
Assuming a higher temperature makes the model more creative. It makes sampling less predictable. On a prompt with one obvious answer it changes nothing, and on an open one it mostly adds noise.
If this comes up in an interview
What is the difference between temperature and top-p?
Temperature rescales the whole probability distribution, making peaks sharper or flatter. top-p truncates it, keeping the smallest set of tokens whose probabilities sum to p and sampling among those. Change one of them, not both, or the interaction becomes hard to reason about.
Is temperature 0 deterministic?
Not on a hosted API. Batched GPU execution, non-associative floating point addition, expert routing and varying hardware all introduce small differences. Locally, with a fixed seed and a single request, it can be.
You need consistent JSON output. What settings do you use?
Low temperature, around 0 to 0.2, and schema-constrained decoding if available, which makes structure a guarantee rather than a preference. Then check the stop reason, because truncation produces invalid JSON regardless of temperature.
Build this: Run one prompt ten times at temperature 0 and ten at 1.5, and count distinct outputs. Then do it with a prompt that has one overwhelmingly likely answer and note that temperature barely moves it.
Watch it explained
This is one section of Sampling & decoding, 9 hours across 3 sections. Read it in full to get the whole topic in order.
Stop 10 of 50 on the AI Engineer route
0 of 50 topics done · 0h behind you, 603h to go
What you have marked done is stored in this browser only. It does not reach a server, and clearing site data will clear it.