Asked about How a model picks the next token
Is temperature 0 deterministic?
An answer that works
Not on a hosted API. Batched GPU execution, non-associative floating point addition, expert routing and varying hardware all introduce small differences. Locally, with a fixed seed and a single request, it can be.
Reading an answer is not being able to give one. The section that teaches this is How a model picks the next token, 3 hours of Sampling & decoding.
Learn this properlyAlso asked about Sampling & decoding
What is the difference between temperature and top-p?You need consistent JSON output. What settings do you use?What is self-consistency?Your answers are sometimes cut off mid-sentence. Why?Implement top-p sampling given a vector of logits.Why is greedy decoding the same as temperature zero?Why use argpartition rather than argsort for top-k?