Asked about Shapes, broadcasting and the bugs they cause
A regression model's loss decreases slowly but predictions are nonsense. What would you check first?
An answer that works
Shapes. A prediction of shape [N, 1] subtracted from targets of shape [N] broadcasts to [N, N] and computes a meaningless loss without error. Assert shapes where the loss is computed, and squeeze or reshape explicitly.
Then they push further
Why does view sometimes fail after transpose?
Transpose only changes strides, so the data is no longer laid out contiguously for the new shape. view cannot reinterpret non-contiguous memory. Call contiguous() first, or use reshape, which copies if it has to.
What gets marked up
Marked up for naming the silent broadcast bug specifically. It is one of the most common real training bugs.
Reading an answer is not being able to give one. The section that teaches this is Shapes, broadcasting and the bugs they cause, 4 hours of PyTorch.
Learn this properlyAlso asked about PyTorch