Asked about Running a fine-tune properly

How do you know the fine-tune actually beat the prompt?

An answer that works

Score both on the identical held-out set with the identical grader in the same run, and keep the pairing. Report the disagreement counts rather than just the two accuracies: 78 against 71 means something different when the disagreements are 28 to 14 than when they are 70 to 63, and only the first is a result. A paired test such as McNemar uses only the examples the two systems disagreed on, which is where all the information about the difference is; throwing the pairing away and comparing two accuracies can turn a significant result into an interval that contains zero on the same data.

What gets marked up

Passable answers name a held-out set. Strong ones insist on the same examples for both systems, on the best prompt rather than a weak one as the baseline, and on reading the cases the prompt got right and the fine-tune got wrong before shipping.

Reading an answer is not being able to give one. The section that teaches this is Running a fine-tune properly, 6 hours of Fine-tuning.

Learn this properly

Also asked about Fine-tuning

When would you fine-tune rather than use RAG?What is LoRA?What is the cost of fine-tuning beyond the training run?What is distillation?How much data do you need to fine-tune?Where do the examples come from?What is catastrophic forgetting?You have five fine-tunes of one base model to serve. Merged or separate adapters?
All 13 questions on Fine-tuning