Asked about Sampling settings for real tasks
Your answers are sometimes cut off mid-sentence. Why?
An answer that works
The response hit max_tokens. The model does not plan around that limit, it is simply stopped. Check the stop reason, raise the limit or ask for shorter output in the prompt.
Reading an answer is not being able to give one. The section that teaches this is Sampling settings for real tasks, 3 hours of Sampling & decoding.
Learn this properlyAlso asked about Sampling & decoding
What is the difference between temperature and top-p?Is temperature 0 deterministic?You need consistent JSON output. What settings do you use?What is self-consistency?Implement top-p sampling given a vector of logits.Why is greedy decoding the same as temperature zero?Why use argpartition rather than argsort for top-k?