Asked about Working on a remote GPU machine
Your training job died overnight. How do you find out why?
An answer that works
Check the job's log for a traceback, then system signs: whether the disk filled, whether the process was killed for running out of memory (the kernel log shows this), and whether the SSH session ended because it was not in tmux. Check the GPU for errors too.
Then they push further
How do you make the next run survive this?
Run in tmux or a job scheduler, save checkpoints regularly and resume from the latest, check disk space before starting, and log to a file rather than only the terminal.
What gets marked up
Marked up for a systematic list of causes and for checkpointing and resuming rather than just rerunning.
Reading an answer is not being able to give one. The section that teaches this is Working on a remote GPU machine, 4 hours of Linux & the shell.
Learn this properly