Asked about Reading papers without reading papers
A paper reports a gain on a benchmark. How do you decide whether it applies to your system?
An answer that works
Compare the setting rather than the number. Is the baseline the thing I actually run, is the data distribution like mine, and is the effect larger than the seed spread they report? If it survives that and it would be cheap to test, I run it on my own eval rather than adopting it, because my eval is the only one I will be judged on.
Reading an answer is not being able to give one. The section that teaches this is Reading papers without reading papers, 5 hours of Reading papers.
Learn this properlyAlso asked about Reading papers
How do you keep up with the literature?What makes you sceptical of a paper's result?When is it worth reimplementing a paper rather than reading it?What do you check first in a paper's results?What makes you doubt a result will replicate?How do you approach a dense equation in a paper you need to implement?You tried to reproduce a paper and got a result 5 points below theirs. What do you do?