Asked about How a model picks the next token
What is the difference between temperature and top-p?
An answer that works
Temperature rescales the whole probability distribution, making peaks sharper or flatter. top-p truncates it, keeping the smallest set of tokens whose probabilities sum to p and sampling among those. Change one of them, not both, or the interaction becomes hard to reason about.
Reading an answer is not being able to give one. The section that teaches this is How a model picks the next token, 3 hours of Sampling & decoding.
Learn this properlyAlso asked about Sampling & decoding
Is temperature 0 deterministic?You need consistent JSON output. What settings do you use?What is self-consistency?Your answers are sometimes cut off mid-sentence. Why?Implement top-p sampling given a vector of logits.Why is greedy decoding the same as temperature zero?Why use argpartition rather than argsort for top-k?