Asked about How preference training is done
What is constitutional AI?
An answer that works
A model critiques and revises its own outputs against written principles, producing preference data without human annotators for every comparison. It scales feedback and moves judgement into an explicit, inspectable specification.
Reading an answer is not being able to give one. The section that teaches this is How preference training is done, 4 hours of Alignment & RLHF.
Learn this properlyAlso asked about Alignment & RLHF
Why can you not use a pretrained model directly as an assistant?What is the difference between RLHF and DPO?Why are models sycophantic?Why use pairwise comparisons for preference data?What is the KL penalty in RLHF for?Why mask the loss on prompt tokens during supervised fine-tuning?What does the beta parameter in DPO control?How would you measure whether a model is sycophantic?