Asked about Sampling settings for real tasks
What is self-consistency?
An answer that works
Sampling several answers to the same question and taking the most common one. Independent errors rarely agree, so the majority answer is more accurate than one sample. It costs one call per sample, so it suits high value questions with a single correct answer.
Then they push further
What does it mean when the samples disagree?
That the model is uncertain on this input. That is useful: route those cases to a person or a stronger model instead of trusting the majority of a split vote.
Why does it not help for open-ended writing?
There is no single answer to vote on. For writing you would need a judge or a person to choose between samples, which is a different technique with its own costs.
What gets marked up
Marked up for naming the cost and for using disagreement between samples as a confidence signal.
Reading an answer is not being able to give one. The section that teaches this is Sampling settings for real tasks, 3 hours of Sampling & decoding.
Learn this properlyAlso asked about Sampling & decoding