Asked about Reading papers without reading papers
When is it worth reimplementing a paper rather than reading it?
An answer that works
When I am about to build on it, and then scoped to one figure at the smallest scale where the effect should still appear, with my own baseline run first. A handful a year. What it teaches is the size of the gap between a paper and a working system: the schedule, the prompt template, the eval harness version and the checkpoint selection are routinely unstated, and after guessing at those once you read every paper differently.
Reading an answer is not being able to give one. The section that teaches this is Reading papers without reading papers, 5 hours of Reading papers.
Learn this properlyAlso asked about Reading papers
How do you keep up with the literature?What makes you sceptical of a paper's result?A paper reports a gain on a benchmark. How do you decide whether it applies to your system?What do you check first in a paper's results?What makes you doubt a result will replicate?How do you approach a dense equation in a paper you need to implement?You tried to reproduce a paper and got a result 5 points below theirs. What do you do?