Asked about Turning a predictor into an assistant
Why can you not use a pretrained model directly as an assistant?
An answer that works
It continues text rather than answering. A question may be continued with more questions, because that is a plausible continuation. Instruction tuning teaches the question-answer shape, and preference methods refine quality beyond that.
Reading an answer is not being able to give one. The section that teaches this is Turning a predictor into an assistant, 5 hours of Alignment & RLHF.
Learn this properlyAlso asked about Alignment & RLHF
What is the difference between RLHF and DPO?Why are models sycophantic?Why use pairwise comparisons for preference data?What is the KL penalty in RLHF for?What is constitutional AI?Why mask the loss on prompt tokens during supervised fine-tuning?What does the beta parameter in DPO control?How would you measure whether a model is sycophantic?