Asked about Autograd, custom pieces and performance
How would you speed up a training run?
An answer that works
Profile first, because the bottleneck is usually data loading or synchronisation rather than arithmetic. Then mixed precision, more data loader workers, removing per-step .item() calls, and torch.compile.
Reading an answer is not being able to give one. The section that teaches this is Autograd, custom pieces and performance, 5 hours of PyTorch.
Learn this properlyAlso asked about PyTorch
What does model.eval() do?Why wrap validation in torch.no_grad()?Your GPU utilisation is 20 percent during training. What is wrong?Why might a tensor's gradient be None after backward?A regression model's loss decreases slowly but predictions are nonsense. What would you check first?Walk me through a minimal GPT-style model.Your model's training loss is flat from the first step. How do you debug it?What do you need to save to resume a training run exactly?