Asked about Checkpoints and reproducibility
What do you need to save to resume a training run exactly?
An answer that works
Model weights, optimiser state, scheduler state, the step or epoch counter, random number generator states, and the position in the data. Missing the optimiser state is the common mistake: Adam restarts with empty moment estimates and training behaves differently.
Reading an answer is not being able to give one. The section that teaches this is Checkpoints and reproducibility, 3 hours of PyTorch.
Learn this properlyAlso asked about PyTorch
What does model.eval() do?Why wrap validation in torch.no_grad()?Your GPU utilisation is 20 percent during training. What is wrong?Why might a tensor's gradient be None after backward?How would you speed up a training run?A regression model's loss decreases slowly but predictions are nonsense. What would you check first?Walk me through a minimal GPT-style model.Your model's training loss is flat from the first step. How do you debug it?