ProgrammingFoundation10h

Linux & the shell

Where the training runs and the servers live.

Linux is the operating system almost every server and GPU machine runs. The shell is the text interface to it: you type a command, it does the thing and prints the result, with no windows and no icons. Commands can be chained so the output of one becomes the input of the next, which is how three short lines can search a million lines of log.

The machines that matter are not your laptop. Training runs on a rented GPU box, production runs in a container, and you reach both over a network connection with nothing but a terminal. When something breaks there, being unable to read a log, list the running processes or see what filled the disk means you cannot work at all.

It is the foundation under containers, cloud and deployment, all of which are ways of running things on Linux machines you never sit in front of. It pairs with debugging, because the evidence you need usually lives in a file on one of those machines.

Every debugging session that matters ends up on a machine with no editor and no browser.

In more detailNavigating a filesystem, piping commands together, reading logs, managing processes and working over SSH. GPUs live on remote Linux boxes and production runs on containers, so the terminal is the only interface you are guaranteed to have.

Section 1 of 2, 6h

The shell and the server

By the end: You can work on a remote machine with no IDE, which is where the GPUs are.

Navigating and finding things
Reading logs and following them live
Processes, ports and what is using them
Permissions and environment variables

Training runs and production services live on machines you reach through a terminal, with no editor and no debugger. This section is the working subset of Linux that makes that bearable: finding files, following a log while it is being written, seeing which process is holding a port, and handling secrets without committing them. Without it, a service failing on a remote box at midnight is something you can only watch.

What you need to know                      What you type
where am I, what is here                   pwd, ls -la
a file by name, anywhere below here        find . -name "*.jsonl"
a string inside files                      grep -rn "timeout" src/
the last part of a log                     tail -n 200 app.log
the log as it is being written             tail -f app.log
which errors, and how many of each         grep ERROR app.log | sort | uniq -c | sort -rn
is the disk full                           df -h
what filled it                             du -h -d 1 /var | sort -h
what is running                            ps aux --sort=-%mem | head
what is holding port 8000                  ss -tlnp | grep 8000
stop it                                    kill PID, then kill -9 PID
the GPU                                    nvidia-smi
what a variable is set to                  echo $API_KEY
which binary a command resolves to         which python

Worth memorising. Everything below is these fifteen lines explained.

Where you are and what is there#

The shell always has a current directory, and every relative path is read from it. pwd prints it, cd changes it, and ls lists it. Four shorthands do most of the navigating: a dot is here, two dots are the directory above, a tilde is your home directory, and a bare cd with no argument goes home.

$ pwd
/home/jack/project

$ ls -la
total 48
drwxr-xr-x  5 jack jack  4096 Mar 11 09:14 .
drwxr-xr-x 12 jack jack  4096 Mar  2 18:02 ..
-rw-------  1 jack jack    84 Mar 11 09:12 .env
drwxr-xr-x  8 jack jack  4096 Mar 11 09:14 .git
-rw-r--r--  1 jack jack  2011 Mar 10 22:40 Dockerfile
-rw-r--r--  1 jack jack 31204 Mar 11 09:03 app.log
drwxr-xr-x  2 jack jack  4096 Mar  9 14:55 src

$ cd src && cd ..          # .. is one level up
$ cd -                     # back to the previous directory

The first column is permissions, the third and fourth are owner and group, then size in bytes, then modified time.

Two habits make this bearable on a machine you do not know. Press Tab to complete a path rather than typing it, because it also confirms the file exists. And read ls -la rather than ls: the dotfiles it hides are usually the ones you are looking for, and .env above is one of them.

Pipes and redirection#

This is the idea the whole shell is built on, and it is worth being precise about. Every command has three channels: standard input, which it reads; standard output, where its results go; and standard error, where its complaints go. By default both outputs land on your screen, which is why they look like one thing. They are not, and most confusion here comes from forgetting it.

command > file        standard output into file, overwriting it
command >> file       standard output appended to file
command 2> file       standard error into file
command > file 2>&1   both into file, error redirected to where output now goes
command < file        file becomes the command's standard input
a | b                 a's standard output becomes b's standard input
command > /dev/null   discard output
command | tee file    to the screen and to a file at once

The order in 'file 2>&1' matters: it means send error where output is already going, so the redirect must come first.

# Run a training job and keep everything, including the traceback.
python train.py > train.log 2>&1

# This one loses the traceback, because 2>&1 is applied before the redirect.
python train.py 2>&1 > train.log        # wrong, and the mistake is invisible

# Watch it and keep it.
python train.py 2>&1 | tee train.log

The middle line is the failure people hit at three in the morning: the log exists and the error is not in it.

A pipe connects programs that each do one thing, which is why a handful of commands compose into an answer. The five that appear in nearly every pipeline are sort, uniq -c, which counts adjacent duplicates and so requires sorted input, wc -l for counting lines, head and tail for the ends, and cut or awk for picking a column out.

Finding files#

find . -name "*.jsonl"                 # by name, recursively, from here
find . -iname "*README*"               # case-insensitive
find . -type d -name "checkpoints"     # directories only
find . -size +1G                       # bigger than a gigabyte
find . -mtime -1                       # modified in the last day
find /var/log -name "*.gz" -delete     # dangerous: read it before you run it

# What the disk is doing, largest first
$ find . -type f -size +100M -exec ls -lh {} \;
-rw-r--r-- 1 jack jack 2.4G Mar  9 11:02 ./runs/exp7/checkpoint-4000.pt
-rw-r--r-- 1 jack jack 2.4G Mar  9 13:44 ./runs/exp7/checkpoint-8000.pt

The braces are the file find matched; the escaped semicolon ends the -exec command. It is ugly and it is worth memorising.

find is recursive and ls is not, which is the single most common reason a beginner concludes a file is not on the machine. ls only shows the directory you name. If you do not know where something is, the tool is find, or locate if the machine has an index built.

Finding text#

grep -rn "OPENAI_API_KEY" .        # -r recursive, -n with line numbers
grep -i error app.log             # case-insensitive
grep -c ERROR app.log             # just the count
grep -v DEBUG app.log             # invert: everything that is not DEBUG
grep -C 3 Traceback app.log       # 3 lines of context either side
grep -E "4[0-9]{2}|5[0-9]{2}" app.log    # extended regex: 4xx and 5xx

$ grep -rn "timeout" src/
src/client.py:14:    TIMEOUT_SECONDS = 30
src/client.py:61:        except httpx.ReadTimeout:
src/worker.py:88:    # TODO: the timeout here should be configurable

-C 3 around a Traceback is the fastest way to read a Python error out of a busy log.

Reading a log#

A production log is too big to read and too important to guess at, so you never open it whole. You look at the end of it, you follow it while something is happening, or you reduce it to a count. Those are three different moves and each has its command.

tail -n 200 app.log                  # the last 200 lines: what just happened
tail -f app.log                      # follow it live; Ctrl-C to stop
tail -f app.log | grep -i error      # follow, but only the errors
less app.log                         # page through it: / to search, G to end, q to quit
journalctl -u myservice -f           # the same, for a systemd service
journalctl -u myservice --since "10 min ago"
docker logs -f --tail 100 mycontainer

less is the one to learn: it opens a two gigabyte file instantly because it does not read all of it.

Then the four one-liners. Between them they answer most questions anyone asks of a log, and the third one, following a single request or run id through the file, is the fastest debugging move available to you.

# 1. Which errors, and how many of each. sort first: uniq only sees neighbours.
$ grep ERROR app.log | cut -d' ' -f4- | sort | uniq -c | sort -rn | head
    412 ERROR upstream timeout after 30s
     97 ERROR rate_limited status=429
     12 ERROR ValidationError: amount must be > 0
      1 ERROR OSError: [Errno 28] No space left on device

# 2. When did it start. Count errors per hour.
$ grep ERROR app.log | cut -c1-13 | uniq -c
     3 2026-03-11T07
     4 2026-03-11T08
   402 2026-03-11T09        <- 09:00 is when to look

# 3. Everything about one request, in order.
grep "req_8f2c" app.log | less

# 4. The busiest callers, when something is being hammered.
grep "POST /generate" app.log | awk '{print $6}' | sort | uniq -c | sort -rn | head -5

sort, uniq -c, sort -rn, head. That four-stage pipeline is the shape of nearly every log question.

uniq -c without sort is the classic wrong answer

uniq collapses adjacent identical lines only. On an unsorted log it reports every run separately, so a message appearing four hundred times scattered through the file comes back as four hundred counts of 1, and the output looks plausible. Always sort before uniq.

Why the disk is full#

Checkpoints, model caches and container images fill disks quietly, and a full disk fails in ways that do not mention the disk: a training run dies at its first save, Postgres refuses writes, a build fails on an unrelated step. Check it first, always, and the diagnosis is two commands.

$ df -h                              # which filesystem, and how full
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        97G   92G  0.5G  99% /
/dev/nvme1n1    1.8T  310G  1.4T  18% /mnt/data

# Then walk down into the full one, one level at a time.
$ du -h -d 1 / 2>/dev/null | sort -h | tail -5
2.1G    /usr
6.4G    /var
11G     /root
68G     /home

$ du -h -d 1 /home/jack | sort -h | tail -3
4.2G    /home/jack/.cache          # huggingface downloads live here
9.1G    /home/jack/project
52G     /home/jack/runs            # found it

-d 1 keeps it to one level so you descend deliberately. sort -h sorts 2.1G above 900M, which plain sort does not.

If df says the disk is full but du cannot find the space, the space is held by a deleted file that a process still has open. Deleting a log that a running service is writing to frees nothing until the service is restarted, because the file only truly goes when the last handle closes.

lsof +L1                 # open files with no name left on disk
docker system df         # on a build machine this is usually the answer
docker system prune -a   # reclaims it, and removes images you may want

Processes, and stopping one#

$ ps aux --sort=-%mem | head -4
USER  PID  %CPU %MEM     VSZ    RSS TTY STAT START TIME COMMAND
jack 4182  99.4 61.2 9812344 9.8G  ?   Rl   09:02 12:41 python train.py --bs 64
jack 3120   0.3  1.1  712004 180M  ?   Sl   08:31  0:04 uvicorn app:api
root  912   0.0  0.2  102884  31M  ?   Ss   Mar02  0:31 /usr/bin/containerd

htop                     # the interactive version, if it is installed
nvidia-smi               # GPU memory and which PID is holding it
free -h                  # RAM, and how much is actually available

RSS is the memory actually resident. VSZ is address space and is usually enormous and meaningless.

To stop something you send it a signal. kill with no options sends TERM, which asks the process to shut down and lets it flush its files and finish its current request. kill -9 sends KILL, which the process cannot catch or handle, so anything half-written stays half-written. Try TERM first and give it a few seconds; reach for -9 only when it has not gone.

kill 4182                # TERM: please stop
kill -9 4182             # KILL: stop now, no cleanup
pkill -f "train.py"      # by command line, when you do not know the PID
Ctrl-C                   # TERM to the thing in your terminal
Ctrl-Z then bg           # suspend it, then resume it in the background
jobs                     # what this shell is running in the background

# Which PID is on port 8000
$ ss -tlnp | grep 8000
LISTEN 0  2048  0.0.0.0:8000  0.0.0.0:*  users:(("uvicorn",pid=3120,fd=8))

lsof -i :8000 does the same as the ss line and is the one that also exists on macOS.

Exit code 137 means the kernel killed it

A container or job that stops with 137 was sent KILL, and almost always by the out-of-memory killer because it asked for more RAM than the machine had. Check with dmesg | grep -i oom, or the kubectl describe output. Retrying changes nothing; the fix is less memory per process or a bigger machine.

Permissions#

The ten characters at the start of ls -la are a file type followed by three groups of read, write and execute, for the owner, the group, and everyone else. The numeric form people type is the same thing: read is 4, write is 2, execute is 1, added up per group.

-rwxr-xr--  1 jack devs  2011 Mar 10 22:40 run.sh
 |                                           owner jack: rwx = 4+2+1 = 7
 |                                           group devs: r-x = 4+0+1 = 5
 |                                           others:     r-- = 4+0+0 = 4
 first character: - file, d directory, l symlink        so: chmod 754 run.sh

chmod +x run.sh      make it runnable
chmod 600 .env       owner may read and write, nobody else may read it at all
chmod 755 script.sh  the normal permission for a script others should run
chown -R jack:devs /srv/app     change owner and group, recursively

chmod 600 on anything holding a key. A world-readable .env on a shared box is a real leak, not a theoretical one.

Permission denied on running a script usually means the execute bit is missing, not that you lack rights; chmod +x fixes it. Permission denied on writing under /usr or /etc means the path belongs to root and needs sudo. Bash: command not found on a script you can see usually means the wrong interpreter line or Windows line endings, not a permissions problem at all.

Environment variables#

export ANTHROPIC_API_KEY=sk-ant-...   # set for this shell and anything it starts
echo $ANTHROPIC_API_KEY               # read it back
env | grep -i api                     # everything set, filtered
unset ANTHROPIC_API_KEY

echo $PATH                            # the directories searched, in order
which python                          # the one that would actually run
# /home/jack/project/.venv/bin/python

API_KEY=test python check.py          # set it for one command only
source .env                           # load a file of KEY=value lines into this shell

which python answers most 'but I installed that package' questions: a different interpreter is first on PATH.

Quoting is the other thing to get right. Double quotes expand variables and single quotes do not, and an unquoted variable containing a space is split into several arguments, which is how a script deletes the wrong thing. Quote every variable that holds a path.

$ name="my runs"
$ echo "$name"     # my runs
$ echo '$name'     # $name
$ rm -r $name      # tries to remove 'my' and 'runs'. Quote it.

Secrets in environment variables, not files in the repo

Environment variables are visible to the process and not committed by accident. A .env file is fine locally if it is gitignored and has restrictive permissions, and should not be how production gets its secrets. Note that ps and /proc expose a process's environment to other users on the same machine, so on a shared box a secret manager is the real answer.

Reading a shell script someone else wrote#

You will read far more shell than you write, usually an entrypoint script in a container or a deploy script in CI. Six pieces of syntax cover almost all of it, and the one that matters most is the line at the top that decides whether the script stops when something fails.

#!/usr/bin/env bash
set -euo pipefail          # read this line first; see below

MODEL_DIR="${MODEL_DIR:-/models}"     # use $MODEL_DIR if set, else /models
PORT="${PORT:-8000}"

if [ ! -d "$MODEL_DIR" ]; then         # -d: does this directory exist
  echo "no model dir at $MODEL_DIR" >&2   # >&2 sends it to standard error
  exit 1
fi

python -m app.download --dest "$MODEL_DIR" \
  && echo "download ok" \
  || { echo "download failed" >&2; exit 1; }

exec uvicorn app:api --host 0.0.0.0 --port "$PORT" "$@"

exec replaces the shell with the server, so signals from Docker reach the server rather than the wrapper. Without it, docker stop hangs for ten seconds and then kills it.

set -e stops the script at the first command that fails. Without it a failed download is followed cheerfully by a server starting with no model.
set -u makes an unset variable an error rather than an empty string, which is what stops rm -rf $DIR/ becoming rm -rf /.
set -o pipefail makes a | b fail if a failed. Without it, a pipeline reports only the last command's status and a broken first stage is invisible.
The dollar-brace-colon-dash form gives a default: it uses the variable if set and non-empty, otherwise the value after the dash.
Double quotes around $@ pass the script's own arguments through, each preserved even if it contains spaces.
&& runs the next command only if this one succeeded; || only if it failed. $? holds the exit status of the last command, where 0 is success.

Where people go wrong

Editing files directly on a production server. The change is invisible to everyone, survives no deploy, and the next release silently reverts it.

Build this: Deploy something to a remote machine and debug it using only the terminal: find the process, read the logs, identify a failure and fix it without opening an editor locally.

Open this section on its own page

Section 2 of 2, 4h

Working on a remote GPU machine

By the end: You can connect to a remote machine, run long jobs that survive disconnection, move data, and watch the GPU.

SSH keys and config
Jobs that survive a dropped connection
Moving files and data
Watching GPU, memory and disk
Running things on a schedule

GPUs are rented, so real training runs happen on a machine somewhere else that you reach over SSH. This section is the working setup for that: keys and a config alias so connecting is one word, tmux so a twelve-hour run does not die when your laptop sleeps, rsync for moving data and results back, and the few commands that tell you whether the GPU, the memory or the disk is about to stop you. A disk filling up and killing a run at its first checkpoint is the most common avoidable loss here.

Connect once, properly#

SSH proves who you are with a key pair rather than a password: a private key that stays on your laptop and never moves, and a public key you install on every machine you want to reach. Make the pair once. Copying the private key to a server, or pasting it into CI, is the mistake that turns one compromised box into all of them.

ssh-keygen -t ed25519            # make a key pair once; ~/.ssh/id_ed25519 and .pub
ssh-copy-id jack@203.0.113.10    # append the .pub to the server's authorized_keys

$ ssh jack@203.0.113.10
The authenticity of host '203.0.113.10' can't be established.
ED25519 key fingerprint is SHA256:0Zn1...  Are you sure you want to continue? yes
jack@gpu-01:~$

Permissions matter here and the error does not say so: ~/.ssh must be 700 and the private key 600, or SSH refuses to use it.

# ~/.ssh/config
Host gpu
  HostName 203.0.113.10
  User jack
  IdentityFile ~/.ssh/id_ed25519
  ServerAliveInterval 60      # keep the connection open through a quiet NAT
  LocalForward 8888 localhost:8888   # a notebook on the box, reachable at localhost:8888

Now 'ssh gpu' is enough, and so is 'scp file gpu:' and 'rsync gpu:runs/ .'. Everything below uses that alias.

ssh gpu                              # interactive shell
ssh gpu "nvidia-smi --query-gpu=memory.used --format=csv"   # one command, then exit
ssh gpu "tail -f ~/runs/exp7/train.log"                     # follow a remote log locally
ssh -L 6006:localhost:6006 gpu       # tunnel TensorBoard to your own browser

Running one command over SSH and piping it locally is how you check twenty machines without logging into any of them.

Port forwarding is what makes a remote notebook usable

The GPU box will not let you open port 8888 to the internet, and it should not. LocalForward tunnels it through the SSH connection you already have, so the notebook appears on your own machine at localhost:8888 with no firewall change and no exposure. The same trick reaches a database or a TensorBoard.

Moving files: scp and rsync#

scp copies a file, once, and is what you want for a single artefact. rsync compares both sides first and sends only what differs, so it is what you want for a directory, for anything large, and for anything you will send again. The distinction that trips everyone is the trailing slash on an rsync source.

scp config.yaml gpu:~/project/            # one file up
scp gpu:~/runs/exp7/metrics.json .        # one file down
scp -r gpu:~/runs/exp7 ./exp7             # a directory, no resume, no skipping

rsync -avP data/ gpu:~/project/data/      # contents of data/ into data/
rsync -avP data  gpu:~/project/           # the data directory itself into project/
rsync -avP gpu:~/project/runs/ ./runs/    # results back
rsync -avP --exclude "*.pt" gpu:~/runs/ ./runs/   # skip the 2 GB checkpoints

$ rsync -avP data/ gpu:~/project/data/
sending incremental file list
train.jsonl
    412,336,128  74%   38.11MB/s    0:00:03

-a preserves timestamps and permissions, -v lists what moved, -P shows progress and resumes a part-sent file. A source ending in a slash means the contents; without it, the directory itself.

Interrupt an rsync and run the same command again: it picks up where it stopped, because it compares before it sends. That is the whole reason to prefer it over scp for a dataset on a connection that might drop.

Jobs that outlive your laptop#

If you start a training run in a plain SSH session and your connection drops, the run dies with it: closing the connection sends a hangup to everything attached to that terminal. tmux breaks that link by running your job in a session that lives on the server and that your terminal merely attaches to. Detach, close the laptop, reconnect from a train, attach again, and the run has been going the whole time.

tmux new -s train        # start a named session, then run the job inside it
# Ctrl-b then d          # detach; the job keeps running on the server
tmux ls                  # what sessions exist
# train: 1 windows (created Wed Mar 11 09:02:41 2026)
tmux attach -t train     # come back later, from anywhere
# Ctrl-b then c          # a new window: one for the run, one for nvidia-smi
# Ctrl-b then n / p      # next and previous window
tmux kill-session -t train

Ctrl-b is the prefix: you press it, release, then press the key. Getting used to that is the only hard part.

nohup python train.py > train.log 2>&1 & is the one-line version if tmux is not installed. It works, and you lose the ability to look at the running process interactively, which is exactly what you want when it starts misbehaving.

Watching the machine#

$ nvidia-smi
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 550.90.07    Driver Version: 550.90.07   CUDA Version: 12.4       |
|  0  NVIDIA A100-SXM4-80GB     On  | 00000000:07:00.0 Off |                0 |
| N/A   64C    P0   312W / 400W |  74216MiB / 81920MiB |     98%      Default |
+-----------------------------------------------------------------------------+
|  0   N/A  N/A      4182      C   python train.py                   74206MiB |
+-----------------------------------------------------------------------------+

watch -n 1 nvidia-smi    # refreshed each second
htop                     # CPU and RAM per process
df -h                    # free disk space
du -sh ~/project/*       # what is using it

Two numbers matter. Memory near the limit means the next batch raises CUDA out of memory. Utilisation near zero while the job runs means the GPU is starving and the bottleneck is your data loading.

Disks fill up

Checkpoints and caches of downloaded models grow quietly. A full disk kills a run at the moment it tries to save, which is the worst possible moment. Check df -h before a long job, keep only the last few checkpoints, and remember that ~/.cache/huggingface is often larger than the project.

When a run does die, the order of checks is fixed: the tail of its own log for a traceback, then df -h for a full disk, then dmesg | grep -i oom for the kernel killing it over memory, then nvidia-smi for a GPU that fell off the bus. A job that vanished with no traceback at all and no disk problem was almost certainly the out-of-memory killer, or an SSH session that was not in tmux.

On a schedule#

# crontab -e      minute hour day month weekday  command
0 3 * * *  /home/jack/project/nightly_eval.sh >> /home/jack/logs/eval.log 2>&1

Redirect both streams to a file. cron has no terminal, so anything printed and not redirected is emailed into the void.

cron has almost no environment

It does not read your .bashrc, so PATH is minimal and your API keys are not set. A script that works when you run it by hand and does nothing at three in the morning is nearly always this. Use absolute paths to every binary, activate the virtualenv inside the script, and source the environment explicitly.

Where people go wrong

Starting a twelve-hour run in a bare SSH session. The laptop sleeps, the connection drops, the run is gone.

Build this: On any remote machine, even a small cloud instance, set up key-based SSH with a config alias, start a long job in tmux, disconnect, reconnect, and sync results back with rsync.

Open this section on its own page

Now actually do it

Reading a topic is not learning it. These are the hours that count, and each one tells you when you are finished.

1Debug a service with only a terminal

Deploy something to a remote machine, break it, and fix it without opening a local editor. Find the process, read the logs, identify the failure, and patch it in place.

Done when: You did the whole cycle over SSH without reaching for an IDE.

2Follow one request through a log

Instrument a service with a request id, generate a hundred requests, then reconstruct one full request's path using only grep and less.

Done when: You can produce the complete story of one request from the log in under a minute.

3Find what is eating the disk and the port

Fill a disk deliberately and find the cause with du. Start two services on the same port and identify the conflict with lsof.

Done when: Both diagnoses took under two minutes and you did not guess.

12 hours of practice

Interview questions

4 questions, grouped by the section that teaches the answer. An answer you have read is not an answer you can give, so use Test yourself and say it out loud first.

The shell and the server

How do you find what is using a port?

lsof -i :PORT, or ss -tlnp on Linux. It gives you the process id, which you can then inspect or kill.

A service is failing on a remote machine. Walk me through what you check.

Whether the process is running, the recent logs filtered for errors, disk and memory, and whether it can reach its dependencies. Then follow a single request id through the log to see where it diverges.

How do you handle secrets on a server?

Environment variables injected by the platform or a secret manager, never committed files. A gitignored .env with restrictive permissions is acceptable locally and not in production.

Working on a remote GPU machine

Your training job died overnight. How do you find out why?

Check the job's log for a traceback, then system signs: whether the disk filled, whether the process was killed for running out of memory (the kernel log shows this), and whether the SSH session ended because it was not in tmux. Check the GPU for errors too.

Then they push further

How do you make the next run survive this?

Run in tmux or a job scheduler, save checkpoints regularly and resume from the latest, check disk space before starting, and log to a file rather than only the terminal.

What gets marked up. Marked up for a systematic list of causes and for checkpointing and resuming rather than just rerunning.

These questions on their own page, one link per question