Asked about Reading a model card and technical report
A new open model claims to beat a larger one on several benchmarks. How do you evaluate the claim?
An answer that works
Check whether the comparison used the same prompts, settings and number of shots, whether contamination was addressed, whether the benchmarks match my use case, and then run my own evaluation on my tasks. Also check the licence and the real usable context length.
Reading an answer is not being able to give one. The section that teaches this is Reading a model card and technical report, 3 hours of Pretraining.
Learn this properlyAlso asked about Pretraining
How does a language model learn?What are scaling laws?Why do models produce confident wrong answers?Roughly what would it cost to pretrain a 7B model?Given a fixed compute budget, how do you choose model size and token count?What matters most for pretraining quality?Why deduplicate a pretraining corpus?Walk me through how a pretraining dataset is built.