Asked about Sampling from scratch in NumPy
Implement top-p sampling given a vector of logits.
An answer that works
Divide the logits by the temperature, sort descending, softmax, take the cumulative sum, and find the first index where it reaches p. Keep everything up to and including that index, renormalise those probabilities so they sum to one, and draw from them. The inclusive cut matters: without it a step where the top token alone exceeds p keeps an empty set.
Then they push further
Why not just use a fixed k?
Because the right number of candidates depends on how confident the model is at that step, and that varies enormously. On a step where one token holds 95% of the mass, a nucleus at 0.9 is one token and k=10 admits nine the model had ruled out. On a genuinely open step the nucleus might be forty tokens and k=10 cuts off thirty the model thought equally plausible. Top-p reads the cut off the distribution instead of fixing it in advance.
Does temperature change which tokens survive top-k?
No, because dividing by a positive number is monotonic, so the ranking is unchanged and the same k tokens survive. It does change the probabilities they are then sampled with. It changes a top-p set, though: flattening the distribution spreads the mass, so more tokens are needed to reach the threshold.
What gets marked up
The inclusive cut and the renormalisation are what get checked. Sorting and cumulative-summing is the easy part; candidates who return an empty set on a confident step, or who forget to renormalise after truncating, are sampling from something that is not a distribution.
Reading an answer is not being able to give one. The section that teaches this is Sampling from scratch in NumPy, 3 hours of Sampling & decoding.
Learn this properlyAlso asked about Sampling & decoding