Asked about Sampling from scratch in NumPy
Why use argpartition rather than argsort for top-k?
An answer that works
Top-k needs the identity of the k largest, not their order, and argpartition finds them in time linear in the vocabulary rather than the n log n a full sort costs. On a vocabulary of a hundred thousand, executed for every token generated, that is a real difference. Nucleus sampling does need the sort, because it depends on the running total in descending order.
Reading an answer is not being able to give one. The section that teaches this is Sampling from scratch in NumPy, 3 hours of Sampling & decoding.
Learn this properlyAlso asked about Sampling & decoding
What is the difference between temperature and top-p?Is temperature 0 deterministic?You need consistent JSON output. What settings do you use?What is self-consistency?Your answers are sometimes cut off mid-sentence. Why?Implement top-p sampling given a vector of logits.Why is greedy decoding the same as temperature zero?