Asked about How a model picks the next token
You need consistent JSON output. What settings do you use?
An answer that works
Low temperature, around 0 to 0.2, and schema-constrained decoding if available, which makes structure a guarantee rather than a preference. Then check the stop reason, because truncation produces invalid JSON regardless of temperature.
Reading an answer is not being able to give one. The section that teaches this is How a model picks the next token, 3 hours of Sampling & decoding.
Learn this properlyAlso asked about Sampling & decoding
What is the difference between temperature and top-p?Is temperature 0 deterministic?What is self-consistency?Your answers are sometimes cut off mid-sentence. Why?Implement top-p sampling given a vector of logits.Why is greedy decoding the same as temperature zero?Why use argpartition rather than argsort for top-k?