Sampling settings for real tasks
By the end: You can pick settings for a task, and use multiple samples on purpose rather than by accident.
- Defaults by kind of task
- max_tokens is a safety limit, not a length control
- Stop sequences
- Sampling several answers and voting
- Best of n with a checker
- Log probabilities, where available
New to sampling & decoding? A model never picks the next token outright. It produces a probability for every token it could say next, and something has to choose one from that spread. Sampling is that choice, and temperature and top-p are the dials on it: low settings make the model take the likeliest option almost every time, higher settings let less likely ones through, which reads as more variety and more invention. Why it matters
Temperature and top-p have sensible defaults for each kind of task, and this section starts there. The larger part is what people get wrong around them. max_tokens does not make a model brief; it cuts it off, often mid-sentence or mid-JSON, which is why you check the stop reason on every response. Stop sequences end generation at a marker and save you paying for text you would throw away. And sampling several answers deliberately — voting between them, or generating a few and picking with a checker — is a real quality lever rather than an accident of a high temperature.
Where beam search went
Beam search keeps several candidate continuations alive at once and returns the highest-scoring whole sequence. It is still standard in translation and speech recognition, where there is one correct output to find. It is not used for chat, because optimising for the most probable sequence produces bland, repetitive text: the safest next word at every step is rarely the interesting one. If you are asked why sampling replaced it, that is the answer.
A starting point by task#
Task Start with
Extraction, classification low temperature, one sample
Code, SQL low temperature, check by running it
Drafting, brainstorming default or higher temperature
Hard reasoning with one answer several samples, then votemax_tokens cuts, it does not shorten#
The model does not know about your max_tokens. It writes as it normally would and is cut off when the limit is reached, often mid-sentence or mid-JSON. Use it as a cost and safety ceiling, check the stop reason on every response, and ask for brevity in the prompt if you want short answers.
Stop sequences#
A stop sequence ends generation as soon as the model produces a given string. It is useful when output has a natural end marker, such as a closing tag, and saves paying for text you would throw away.
Several samples, then vote#
For a question with one right answer, such as a number or a category, you can sample several answers at a normal temperature and take the most common one. Independent mistakes rarely agree, so the majority is more often right than a single sample. The technique is known as self-consistency. It multiplies cost by the number of samples, so use it where accuracy is worth that.
from collections import Counter
answers = [ask(question, temperature=0.8) for _ in range(5)]
best, votes = Counter(answers).most_common(1)[0]
if votes < 3:
flag_for_review(question, answers) # disagreement is itself a signalBest of n, with a checker#
When you can check an answer, generate several and keep one that passes. Code that must compile and pass tests, SQL that must run and return the expected columns. The checker does the hard work and the samples just give it more to choose from.
Log probabilities#
Some APIs can return the probability the model assigned to each token. For classification with short labels, that gives a rough confidence you can threshold. Treat it with care: these probabilities are often overconfident, so calibrate against labelled data before relying on them.
About seeds
Some providers accept a seed to make sampling repeatable and describe it as best effort. It reduces variation, but do not build anything that breaks if two calls differ.
Where people go wrong
Lowering max_tokens to get shorter answers. You get the same answers, cut off.
If this comes up in an interview
What is self-consistency?
Sampling several answers to the same question and taking the most common one. Independent errors rarely agree, so the majority answer is more accurate than one sample. It costs one call per sample, so it suits high value questions with a single correct answer.
Then they push further
What does it mean when the samples disagree?
That the model is uncertain on this input. That is useful: route those cases to a person or a stronger model instead of trusting the majority of a split vote.
Why does it not help for open-ended writing?
There is no single answer to vote on. For writing you would need a judge or a person to choose between samples, which is a different technique with its own costs.
What gets marked up. Marked up for naming the cost and for using disagreement between samples as a confidence signal.
Your answers are sometimes cut off mid-sentence. Why?
The response hit max_tokens. The model does not plan around that limit, it is simply stopped. Check the stop reason, raise the limit or ask for shorter output in the prompt.
Build this: Pick twenty questions with numeric answers. Compare one sample at low temperature with a five-sample majority vote on accuracy and cost, and note how often a split vote marked a wrong answer.
This is one section of Sampling & decoding, 9 hours across 3 sections. Read it in full to get the whole topic in order.
Stop 10 of 50 on the AI Engineer route
0 of 50 topics done · 0h behind you, 603h to go
What you have marked done is stored in this browser only. It does not reach a server, and clearing site data will clear it.