Sampling & decoding

A model does not decide on an answer. At each step it produces a probability for every token it could write next, and something has to pick one of them. That picking is sampling, and the settings you have seen copied into every code example, temperature, top-p and top-k, are all ways of reshaping those probabilities before the draw. This section is about what each one actually does, and why even the most deterministic setting on offer will not guarantee the same output twice.

Every step is a choice#

At each step the model produces a probability for every token in its vocabulary. Sampling is how one of them gets picked. All the settings you can change are ways of reshaping that distribution before the draw.

After "The capital of France is":

  " Paris"    0.94
  " located"  0.02
  " a"        0.01
  ...everything else shares the remaining 0.03

Temperature#

Temperature divides the raw scores, the logits, before they are turned into probabilities. Below 1 the peaks get sharper and the likely token becomes almost certain. Above 1 the distribution flattens and unlikely tokens get a real chance.

Original:    Paris 0.94   located 0.02   a 0.01

t = 0.2:     Paris 0.99+  everything else vanishes
t = 1.0:     unchanged
t = 1.8:     Paris 0.60   others rise substantially

top-p and top-k#

These truncate rather than rescale. top-k keeps the k most likely tokens and discards the rest. top-p, also called nucleus sampling, keeps the smallest set of tokens whose probabilities add up to p.

top_p = 0.9   keep tokens until their probabilities sum to 0.9,
              then sample among those

Adaptive: when the model is confident, that is one or two tokens.
When it is unsure, it is many.

In practice, change one of them. Tuning temperature and top-p together makes the effect of each hard to reason about. Most people leave top-p at its default and adjust temperature.

Choosing#

• Extraction, classification, anything parsed, anything evaluated: 0 to 0.3.
• General assistant output: around 0.7, which is usually the default.
• Creative writing and brainstorming: 0.8 to 1.0.
• Above roughly 1.5: degrades quickly and rarely useful.

Zero is not deterministic#

Temperature 0 means always take the most likely token — greedy decoding — which sounds reproducible. On a hosted API it is not. Requests are batched onto GPUs, floating point addition is not exactly associative so the order of a batched reduction changes the last bits, mixture-of-experts routing can depend on batch composition, and different requests land on different hardware.

"Name any animal", temperature 0, five runs:

  Elephant, Lion, Lion, Lion, Elephant

Same prompt, same setting, different answers.

Temperature and top-p have sensible defaults for each kind of task, and this section starts there. The larger part is what people get wrong around them. max_tokens does not make a model brief; it cuts it off, often mid-sentence or mid-JSON, which is why you check the stop reason on every response. Stop sequences end generation at a marker and save you paying for text you would throw away. And sampling several answers deliberately — voting between them, or generating a few and picking with a checker — is a real quality lever rather than an accident of a high temperature.

A starting point by task#

Task                              Start with
Extraction, classification        low temperature, one sample
Code, SQL                         low temperature, check by running it
Drafting, brainstorming           default or higher temperature
Hard reasoning with one answer    several samples, then vote

max_tokens cuts, it does not shorten#

The model does not know about your max_tokens. It writes as it normally would and is cut off when the limit is reached, often mid-sentence or mid-JSON. Use it as a cost and safety ceiling, check the stop reason on every response, and ask for brevity in the prompt if you want short answers.

Stop sequences#

A stop sequence ends generation as soon as the model produces a given string. It is useful when output has a natural end marker, such as a closing tag, and saves paying for text you would throw away.

Several samples, then vote#

For a question with one right answer, such as a number or a category, you can sample several answers at a normal temperature and take the most common one. Independent mistakes rarely agree, so the majority is more often right than a single sample. The technique is known as self-consistency. It multiplies cost by the number of samples, so use it where accuracy is worth that.

from collections import Counter

answers = [ask(question, temperature=0.8) for _ in range(5)]
best, votes = Counter(answers).most_common(1)[0]
if votes < 3:
    flag_for_review(question, answers)   # disagreement is itself a signal

Best of n, with a checker#

When you can check an answer, generate several and keep one that passes. Code that must compile and pass tests, SQL that must run and return the expected columns. The checker does the hard work and the samples just give it more to choose from.

Log probabilities#

Some APIs can return the probability the model assigned to each token. For classification with short labels, that gives a rough confidence you can threshold. Treat it with care: these probabilities are often overconfident, so calibrate against labelled data before relying on them.

You already know what temperature and top-p do to the output of an API. This section writes them, because the from-scratch version is a common warm-up question and because implementing nucleus sampling is the shortest way to understand the argument for it. Everything here operates on one vector of logits — the raw scores the model produces for every token in the vocabulary, before anything turns them into probabilities.

The softmax and greedy decoding#

Logits are unnormalised and can be any real number. The softmax turns them into a distribution; greedy decoding skips it entirely, because the largest logit is the largest probability and argmax does not care about the monotonic transformation in between.

import numpy as np


def softmax(logits):
    z = logits - logits.max()        # stability: the largest exponent becomes exp(0)
    e = np.exp(z)
    return e / e.sum()


def greedy(logits):
    return int(np.argmax(logits))    # no softmax needed: argmax is unchanged by it

Temperature#

Temperature divides the logits before the softmax. Below one, the gaps between logits widen and the distribution sharpens; above one they narrow and it flattens. In the limit as temperature goes to zero the distribution becomes one-hot on the argmax, which is why temperature zero and greedy are the same operation — and why a real implementation special-cases it rather than dividing by zero.

def sample_temperature(logits, temperature, rng):
    if temperature == 0.0:
        return greedy(logits)        # the limit, not a division
    p = softmax(logits / temperature)
    return int(rng.choice(len(p), p=p))

Note what temperature cannot do. It rescales the gaps between logits, so where one token is already far ahead of the rest, even a high temperature leaves it far ahead. Variety appears only where the model was genuinely undecided.

Top-k#

Top-k keeps the k highest-scoring tokens, renormalises over them and samples. The point is to cut off the long tail of tokens with tiny but nonzero probability, which collectively hold enough mass to be chosen more often than you would like over a few hundred steps. Use argpartition rather than a full sort: it finds the k largest in linear time without ordering them, and order does not matter here.

def sample_top_k(logits, k, temperature, rng):
    logits = logits / temperature
    # the k largest, unordered - cheaper than argsort over a 100k vocabulary
    keep = np.argpartition(logits, -k)[-k:]
    p = softmax(logits[keep])        # renormalise over the survivors only
    return int(rng.choice(keep, p=p))

Nucleus sampling#

Top-p keeps the smallest set of tokens whose probabilities add up to at least p, then renormalises over that set. The size of the set is not fixed: it is whatever the model's confidence makes it. Sort descending, take the cumulative sum, and cut at the first index where the total crosses the threshold — including that index, so the set always has at least one member even when the top token alone exceeds p.

def sample_top_p(logits, top_p, temperature, rng):
    logits = logits / temperature
    order = np.argsort(logits)[::-1]          # descending by probability
    p = softmax(logits[order])
    cum = np.cumsum(p)
    # first index where the running total reaches the threshold, kept inclusively
    cut = int(np.searchsorted(cum, top_p)) + 1
    keep, p = order[:cut], p[:cut]
    return int(rng.choice(keep, p=p / p.sum()))

Why top-p adapts and top-k does not#

The argument is easier to run than to read. Take two logit vectors over a vocabulary of fifty: one peaked, where a single token holds 95% of the mass, and one flat, where the top token holds under 4%.

def nucleus_size(logits, top_p, temperature=1.0):
    p = np.sort(softmax(logits / temperature))[::-1]
    return int(np.searchsorted(np.cumsum(p), top_p)) + 1


rng = np.random.default_rng(0)

peaked = np.full(50, -6.0)
peaked[7], peaked[3] = 6.0, 3.0           # one clear winner, one runner-up
flat = rng.normal(size=50) * 0.4          # no token much ahead of any other

print(nucleus_size(peaked, 0.9))          # 1
print(nucleus_size(flat, 0.9))            # 42

Now set k to ten and ask what it does on each. On the peaked step it admits nine tokens the model had all but ruled out, one of which will eventually be drawn; that is the mechanism behind a model producing a fluent sentence with one word that does not belong. On the flat step it discards thirty-two tokens the model considered roughly as plausible as the ones it kept, which flattens variety precisely where variety was warranted.

with k = 10

peaked step:  top-k keeps 10  |  top-p(0.9) keeps  1
flat   step:  top-k keeps 10  |  top-p(0.9) keeps 42

Two checks worth running#

Sampling code is easy to get subtly wrong and hard to notice, because wrong output still looks like text. Both of these are cheap.

# temperature approaching zero must agree with greedy
assert all(sample_temperature(flat, 1e-6, rng) == greedy(flat) for _ in range(20))

# top-k with k = vocabulary size must reproduce the full distribution
draws = [sample_top_k(flat, 50, 1.0, rng) for _ in range(5000)]
empirical = np.bincount(draws, minlength=50) / 5000
assert np.abs(empirical - softmax(flat)).max() < 0.02