Working on a remote GPU machine
By the end: You can connect to a remote machine, run long jobs that survive disconnection, move data, and watch the GPU.
- SSH keys and config
- Jobs that survive a dropped connection
- Moving files and data
- Watching GPU, memory and disk
- Running things on a schedule
New to linux & the shell? Linux is the operating system almost every server and GPU machine runs. The shell is the text interface to it: you type a command, it does the thing and prints the result, with no windows and no icons. Commands can be chained so the output of one becomes the input of the next, which is how three short lines can search a million lines of log. Why it matters
GPUs are rented, so real training runs happen on a machine somewhere else that you reach over SSH. This section is the working setup for that: keys and a config alias so connecting is one word, tmux so a twelve-hour run does not die when your laptop sleeps, rsync for moving data and results back, and the few commands that tell you whether the GPU, the memory or the disk is about to stop you. A disk filling up and killing a run at its first checkpoint is the most common avoidable loss here.
Connect once, properly#
SSH proves who you are with a key pair rather than a password: a private key that stays on your laptop and never moves, and a public key you install on every machine you want to reach. Make the pair once. Copying the private key to a server, or pasting it into CI, is the mistake that turns one compromised box into all of them.
ssh-keygen -t ed25519 # make a key pair once; ~/.ssh/id_ed25519 and .pub
ssh-copy-id jack@203.0.113.10 # append the .pub to the server's authorized_keys
$ ssh jack@203.0.113.10
The authenticity of host '203.0.113.10' can't be established.
ED25519 key fingerprint is SHA256:0Zn1... Are you sure you want to continue? yes
jack@gpu-01:~$# ~/.ssh/config
Host gpu
HostName 203.0.113.10
User jack
IdentityFile ~/.ssh/id_ed25519
ServerAliveInterval 60 # keep the connection open through a quiet NAT
LocalForward 8888 localhost:8888 # a notebook on the box, reachable at localhost:8888ssh gpu # interactive shell
ssh gpu "nvidia-smi --query-gpu=memory.used --format=csv" # one command, then exit
ssh gpu "tail -f ~/runs/exp7/train.log" # follow a remote log locally
ssh -L 6006:localhost:6006 gpu # tunnel TensorBoard to your own browserPort forwarding is what makes a remote notebook usable
The GPU box will not let you open port 8888 to the internet, and it should not. LocalForward tunnels it through the SSH connection you already have, so the notebook appears on your own machine at localhost:8888 with no firewall change and no exposure. The same trick reaches a database or a TensorBoard.
Moving files: scp and rsync#
scp copies a file, once, and is what you want for a single artefact. rsync compares both sides first and sends only what differs, so it is what you want for a directory, for anything large, and for anything you will send again. The distinction that trips everyone is the trailing slash on an rsync source.
scp config.yaml gpu:~/project/ # one file up
scp gpu:~/runs/exp7/metrics.json . # one file down
scp -r gpu:~/runs/exp7 ./exp7 # a directory, no resume, no skipping
rsync -avP data/ gpu:~/project/data/ # contents of data/ into data/
rsync -avP data gpu:~/project/ # the data directory itself into project/
rsync -avP gpu:~/project/runs/ ./runs/ # results back
rsync -avP --exclude "*.pt" gpu:~/runs/ ./runs/ # skip the 2 GB checkpoints
$ rsync -avP data/ gpu:~/project/data/
sending incremental file list
train.jsonl
412,336,128 74% 38.11MB/s 0:00:03Interrupt an rsync and run the same command again: it picks up where it stopped, because it compares before it sends. That is the whole reason to prefer it over scp for a dataset on a connection that might drop.
Jobs that outlive your laptop#
If you start a training run in a plain SSH session and your connection drops, the run dies with it: closing the connection sends a hangup to everything attached to that terminal. tmux breaks that link by running your job in a session that lives on the server and that your terminal merely attaches to. Detach, close the laptop, reconnect from a train, attach again, and the run has been going the whole time.
tmux new -s train # start a named session, then run the job inside it
# Ctrl-b then d # detach; the job keeps running on the server
tmux ls # what sessions exist
# train: 1 windows (created Wed Mar 11 09:02:41 2026)
tmux attach -t train # come back later, from anywhere
# Ctrl-b then c # a new window: one for the run, one for nvidia-smi
# Ctrl-b then n / p # next and previous window
tmux kill-session -t trainnohup python train.py > train.log 2>&1 & is the one-line version if tmux is not installed. It works, and you lose the ability to look at the running process interactively, which is exactly what you want when it starts misbehaving.
Watching the machine#
$ nvidia-smi
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 550.90.07 Driver Version: 550.90.07 CUDA Version: 12.4 |
| 0 NVIDIA A100-SXM4-80GB On | 00000000:07:00.0 Off | 0 |
| N/A 64C P0 312W / 400W | 74216MiB / 81920MiB | 98% Default |
+-----------------------------------------------------------------------------+
| 0 N/A N/A 4182 C python train.py 74206MiB |
+-----------------------------------------------------------------------------+
watch -n 1 nvidia-smi # refreshed each second
htop # CPU and RAM per process
df -h # free disk space
du -sh ~/project/* # what is using itDisks fill up
Checkpoints and caches of downloaded models grow quietly. A full disk kills a run at the moment it tries to save, which is the worst possible moment. Check df -h before a long job, keep only the last few checkpoints, and remember that ~/.cache/huggingface is often larger than the project.
When a run does die, the order of checks is fixed: the tail of its own log for a traceback, then df -h for a full disk, then dmesg | grep -i oom for the kernel killing it over memory, then nvidia-smi for a GPU that fell off the bus. A job that vanished with no traceback at all and no disk problem was almost certainly the out-of-memory killer, or an SSH session that was not in tmux.
On a schedule#
# crontab -e minute hour day month weekday command
0 3 * * * /home/jack/project/nightly_eval.sh >> /home/jack/logs/eval.log 2>&1cron has almost no environment
It does not read your .bashrc, so PATH is minimal and your API keys are not set. A script that works when you run it by hand and does nothing at three in the morning is nearly always this. Use absolute paths to every binary, activate the virtualenv inside the script, and source the environment explicitly.
Where people go wrong
Starting a twelve-hour run in a bare SSH session. The laptop sleeps, the connection drops, the run is gone.
If this comes up in an interview
Your training job died overnight. How do you find out why?
Check the job's log for a traceback, then system signs: whether the disk filled, whether the process was killed for running out of memory (the kernel log shows this), and whether the SSH session ended because it was not in tmux. Check the GPU for errors too.
Then they push further
How do you make the next run survive this?
Run in tmux or a job scheduler, save checkpoints regularly and resume from the latest, check disk space before starting, and log to a file rather than only the terminal.
What gets marked up. Marked up for a systematic list of causes and for checkpointing and resuming rather than just rerunning.
Build this: On any remote machine, even a small cloud instance, set up key-based SSH with a config alias, start a long job in tmux, disconnect, reconnect, and sync results back with rsync.
Watch it explained
This is one section of Linux & the shell, 10 hours across 2 sections. Read it in full to get the whole topic in order.
Stop 40 of 50 on the AI Engineer route
0 of 50 topics done · 0h behind you, 603h to go
What you have marked done is stored in this browser only. It does not reach a server, and clearing site data will clear it.