Asked about How preference training is done
Why use pairwise comparisons for preference data?
An answer that works
People rank far more consistently than they assign absolute scores, so comparisons produce cleaner training signal. The quality of that data bounds the whole post-training pipeline.
Reading an answer is not being able to give one. The section that teaches this is How preference training is done, 4 hours of Alignment & RLHF.
Learn this properlyAlso asked about Alignment & RLHF
Why can you not use a pretrained model directly as an assistant?What is the difference between RLHF and DPO?Why are models sycophantic?What is the KL penalty in RLHF for?What is constitutional AI?Why mask the loss on prompt tokens during supervised fine-tuning?What does the beta parameter in DPO control?How would you measure whether a model is sycophantic?