The shell and the server
By the end: You can work on a remote machine with no IDE, which is where the GPUs are.
- Navigating and finding things
- Reading logs and following them live
- Processes, ports and what is using them
- Permissions and environment variables
New to linux & the shell? Linux is the operating system almost every server and GPU machine runs. The shell is the text interface to it: you type a command, it does the thing and prints the result, with no windows and no icons. Commands can be chained so the output of one becomes the input of the next, which is how three short lines can search a million lines of log. Why it matters
Training runs and production services live on machines you reach through a terminal, with no editor and no debugger. This section is the working subset of Linux that makes that bearable: finding files, following a log while it is being written, seeing which process is holding a port, and handling secrets without committing them. Without it, a service failing on a remote box at midnight is something you can only watch.
What you need to know What you type
where am I, what is here pwd, ls -la
a file by name, anywhere below here find . -name "*.jsonl"
a string inside files grep -rn "timeout" src/
the last part of a log tail -n 200 app.log
the log as it is being written tail -f app.log
which errors, and how many of each grep ERROR app.log | sort | uniq -c | sort -rn
is the disk full df -h
what filled it du -h -d 1 /var | sort -h
what is running ps aux --sort=-%mem | head
what is holding port 8000 ss -tlnp | grep 8000
stop it kill PID, then kill -9 PID
the GPU nvidia-smi
what a variable is set to echo $API_KEY
which binary a command resolves to which pythonWhere you are and what is there#
The shell always has a current directory, and every relative path is read from it. pwd prints it, cd changes it, and ls lists it. Four shorthands do most of the navigating: a dot is here, two dots are the directory above, a tilde is your home directory, and a bare cd with no argument goes home.
$ pwd
/home/jack/project
$ ls -la
total 48
drwxr-xr-x 5 jack jack 4096 Mar 11 09:14 .
drwxr-xr-x 12 jack jack 4096 Mar 2 18:02 ..
-rw------- 1 jack jack 84 Mar 11 09:12 .env
drwxr-xr-x 8 jack jack 4096 Mar 11 09:14 .git
-rw-r--r-- 1 jack jack 2011 Mar 10 22:40 Dockerfile
-rw-r--r-- 1 jack jack 31204 Mar 11 09:03 app.log
drwxr-xr-x 2 jack jack 4096 Mar 9 14:55 src
$ cd src && cd .. # .. is one level up
$ cd - # back to the previous directoryTwo habits make this bearable on a machine you do not know. Press Tab to complete a path rather than typing it, because it also confirms the file exists. And read ls -la rather than ls: the dotfiles it hides are usually the ones you are looking for, and .env above is one of them.
Pipes and redirection#
This is the idea the whole shell is built on, and it is worth being precise about. Every command has three channels: standard input, which it reads; standard output, where its results go; and standard error, where its complaints go. By default both outputs land on your screen, which is why they look like one thing. They are not, and most confusion here comes from forgetting it.
command > file standard output into file, overwriting it
command >> file standard output appended to file
command 2> file standard error into file
command > file 2>&1 both into file, error redirected to where output now goes
command < file file becomes the command's standard input
a | b a's standard output becomes b's standard input
command > /dev/null discard output
command | tee file to the screen and to a file at once# Run a training job and keep everything, including the traceback.
python train.py > train.log 2>&1
# This one loses the traceback, because 2>&1 is applied before the redirect.
python train.py 2>&1 > train.log # wrong, and the mistake is invisible
# Watch it and keep it.
python train.py 2>&1 | tee train.logA pipe connects programs that each do one thing, which is why a handful of commands compose into an answer. The five that appear in nearly every pipeline are sort, uniq -c, which counts adjacent duplicates and so requires sorted input, wc -l for counting lines, head and tail for the ends, and cut or awk for picking a column out.
Finding files#
find . -name "*.jsonl" # by name, recursively, from here
find . -iname "*README*" # case-insensitive
find . -type d -name "checkpoints" # directories only
find . -size +1G # bigger than a gigabyte
find . -mtime -1 # modified in the last day
find /var/log -name "*.gz" -delete # dangerous: read it before you run it
# What the disk is doing, largest first
$ find . -type f -size +100M -exec ls -lh {} \;
-rw-r--r-- 1 jack jack 2.4G Mar 9 11:02 ./runs/exp7/checkpoint-4000.pt
-rw-r--r-- 1 jack jack 2.4G Mar 9 13:44 ./runs/exp7/checkpoint-8000.ptfind is recursive and ls is not, which is the single most common reason a beginner concludes a file is not on the machine. ls only shows the directory you name. If you do not know where something is, the tool is find, or locate if the machine has an index built.
Finding text#
grep -rn "OPENAI_API_KEY" . # -r recursive, -n with line numbers
grep -i error app.log # case-insensitive
grep -c ERROR app.log # just the count
grep -v DEBUG app.log # invert: everything that is not DEBUG
grep -C 3 Traceback app.log # 3 lines of context either side
grep -E "4[0-9]{2}|5[0-9]{2}" app.log # extended regex: 4xx and 5xx
$ grep -rn "timeout" src/
src/client.py:14: TIMEOUT_SECONDS = 30
src/client.py:61: except httpx.ReadTimeout:
src/worker.py:88: # TODO: the timeout here should be configurableReading a log#
A production log is too big to read and too important to guess at, so you never open it whole. You look at the end of it, you follow it while something is happening, or you reduce it to a count. Those are three different moves and each has its command.
tail -n 200 app.log # the last 200 lines: what just happened
tail -f app.log # follow it live; Ctrl-C to stop
tail -f app.log | grep -i error # follow, but only the errors
less app.log # page through it: / to search, G to end, q to quit
journalctl -u myservice -f # the same, for a systemd service
journalctl -u myservice --since "10 min ago"
docker logs -f --tail 100 mycontainerThen the four one-liners. Between them they answer most questions anyone asks of a log, and the third one, following a single request or run id through the file, is the fastest debugging move available to you.
# 1. Which errors, and how many of each. sort first: uniq only sees neighbours.
$ grep ERROR app.log | cut -d' ' -f4- | sort | uniq -c | sort -rn | head
412 ERROR upstream timeout after 30s
97 ERROR rate_limited status=429
12 ERROR ValidationError: amount must be > 0
1 ERROR OSError: [Errno 28] No space left on device
# 2. When did it start. Count errors per hour.
$ grep ERROR app.log | cut -c1-13 | uniq -c
3 2026-03-11T07
4 2026-03-11T08
402 2026-03-11T09 <- 09:00 is when to look
# 3. Everything about one request, in order.
grep "req_8f2c" app.log | less
# 4. The busiest callers, when something is being hammered.
grep "POST /generate" app.log | awk '{print $6}' | sort | uniq -c | sort -rn | head -5uniq -c without sort is the classic wrong answer
uniq collapses adjacent identical lines only. On an unsorted log it reports every run separately, so a message appearing four hundred times scattered through the file comes back as four hundred counts of 1, and the output looks plausible. Always sort before uniq.
Why the disk is full#
Checkpoints, model caches and container images fill disks quietly, and a full disk fails in ways that do not mention the disk: a training run dies at its first save, Postgres refuses writes, a build fails on an unrelated step. Check it first, always, and the diagnosis is two commands.
$ df -h # which filesystem, and how full
Filesystem Size Used Avail Use% Mounted on
/dev/root 97G 92G 0.5G 99% /
/dev/nvme1n1 1.8T 310G 1.4T 18% /mnt/data
# Then walk down into the full one, one level at a time.
$ du -h -d 1 / 2>/dev/null | sort -h | tail -5
2.1G /usr
6.4G /var
11G /root
68G /home
$ du -h -d 1 /home/jack | sort -h | tail -3
4.2G /home/jack/.cache # huggingface downloads live here
9.1G /home/jack/project
52G /home/jack/runs # found itIf df says the disk is full but du cannot find the space, the space is held by a deleted file that a process still has open. Deleting a log that a running service is writing to frees nothing until the service is restarted, because the file only truly goes when the last handle closes.
lsof +L1 # open files with no name left on disk
docker system df # on a build machine this is usually the answer
docker system prune -a # reclaims it, and removes images you may wantProcesses, and stopping one#
$ ps aux --sort=-%mem | head -4
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
jack 4182 99.4 61.2 9812344 9.8G ? Rl 09:02 12:41 python train.py --bs 64
jack 3120 0.3 1.1 712004 180M ? Sl 08:31 0:04 uvicorn app:api
root 912 0.0 0.2 102884 31M ? Ss Mar02 0:31 /usr/bin/containerd
htop # the interactive version, if it is installed
nvidia-smi # GPU memory and which PID is holding it
free -h # RAM, and how much is actually availableTo stop something you send it a signal. kill with no options sends TERM, which asks the process to shut down and lets it flush its files and finish its current request. kill -9 sends KILL, which the process cannot catch or handle, so anything half-written stays half-written. Try TERM first and give it a few seconds; reach for -9 only when it has not gone.
kill 4182 # TERM: please stop
kill -9 4182 # KILL: stop now, no cleanup
pkill -f "train.py" # by command line, when you do not know the PID
Ctrl-C # TERM to the thing in your terminal
Ctrl-Z then bg # suspend it, then resume it in the background
jobs # what this shell is running in the background
# Which PID is on port 8000
$ ss -tlnp | grep 8000
LISTEN 0 2048 0.0.0.0:8000 0.0.0.0:* users:(("uvicorn",pid=3120,fd=8))Exit code 137 means the kernel killed it
A container or job that stops with 137 was sent KILL, and almost always by the out-of-memory killer because it asked for more RAM than the machine had. Check with dmesg | grep -i oom, or the kubectl describe output. Retrying changes nothing; the fix is less memory per process or a bigger machine.
Permissions#
The ten characters at the start of ls -la are a file type followed by three groups of read, write and execute, for the owner, the group, and everyone else. The numeric form people type is the same thing: read is 4, write is 2, execute is 1, added up per group.
-rwxr-xr-- 1 jack devs 2011 Mar 10 22:40 run.sh
| owner jack: rwx = 4+2+1 = 7
| group devs: r-x = 4+0+1 = 5
| others: r-- = 4+0+0 = 4
first character: - file, d directory, l symlink so: chmod 754 run.sh
chmod +x run.sh make it runnable
chmod 600 .env owner may read and write, nobody else may read it at all
chmod 755 script.sh the normal permission for a script others should run
chown -R jack:devs /srv/app change owner and group, recursivelyPermission denied on running a script usually means the execute bit is missing, not that you lack rights; chmod +x fixes it. Permission denied on writing under /usr or /etc means the path belongs to root and needs sudo. Bash: command not found on a script you can see usually means the wrong interpreter line or Windows line endings, not a permissions problem at all.
Environment variables#
export ANTHROPIC_API_KEY=sk-ant-... # set for this shell and anything it starts
echo $ANTHROPIC_API_KEY # read it back
env | grep -i api # everything set, filtered
unset ANTHROPIC_API_KEY
echo $PATH # the directories searched, in order
which python # the one that would actually run
# /home/jack/project/.venv/bin/python
API_KEY=test python check.py # set it for one command only
source .env # load a file of KEY=value lines into this shellQuoting is the other thing to get right. Double quotes expand variables and single quotes do not, and an unquoted variable containing a space is split into several arguments, which is how a script deletes the wrong thing. Quote every variable that holds a path.
$ name="my runs"
$ echo "$name" # my runs
$ echo '$name' # $name
$ rm -r $name # tries to remove 'my' and 'runs'. Quote it.Secrets in environment variables, not files in the repo
Environment variables are visible to the process and not committed by accident. A .env file is fine locally if it is gitignored and has restrictive permissions, and should not be how production gets its secrets. Note that ps and /proc expose a process's environment to other users on the same machine, so on a shared box a secret manager is the real answer.
Reading a shell script someone else wrote#
You will read far more shell than you write, usually an entrypoint script in a container or a deploy script in CI. Six pieces of syntax cover almost all of it, and the one that matters most is the line at the top that decides whether the script stops when something fails.
#!/usr/bin/env bash
set -euo pipefail # read this line first; see below
MODEL_DIR="${MODEL_DIR:-/models}" # use $MODEL_DIR if set, else /models
PORT="${PORT:-8000}"
if [ ! -d "$MODEL_DIR" ]; then # -d: does this directory exist
echo "no model dir at $MODEL_DIR" >&2 # >&2 sends it to standard error
exit 1
fi
python -m app.download --dest "$MODEL_DIR" \
&& echo "download ok" \
|| { echo "download failed" >&2; exit 1; }
exec uvicorn app:api --host 0.0.0.0 --port "$PORT" "$@"- set -e stops the script at the first command that fails. Without it a failed download is followed cheerfully by a server starting with no model.
- set -u makes an unset variable an error rather than an empty string, which is what stops rm -rf $DIR/ becoming rm -rf /.
- set -o pipefail makes a | b fail if a failed. Without it, a pipeline reports only the last command's status and a broken first stage is invisible.
- The dollar-brace-colon-dash form gives a default: it uses the variable if set and non-empty, otherwise the value after the dash.
- Double quotes around $@ pass the script's own arguments through, each preserved even if it contains spaces.
- && runs the next command only if this one succeeded; || only if it failed. $? holds the exit status of the last command, where 0 is success.
Where people go wrong
Editing files directly on a production server. The change is invisible to everyone, survives no deploy, and the next release silently reverts it.
If this comes up in an interview
How do you find what is using a port?
lsof -i :PORT, or ss -tlnp on Linux. It gives you the process id, which you can then inspect or kill.
A service is failing on a remote machine. Walk me through what you check.
Whether the process is running, the recent logs filtered for errors, disk and memory, and whether it can reach its dependencies. Then follow a single request id through the log to see where it diverges.
How do you handle secrets on a server?
Environment variables injected by the platform or a secret manager, never committed files. A gitignored .env with restrictive permissions is acceptable locally and not in production.
Build this: Deploy something to a remote machine and debug it using only the terminal: find the process, read the logs, identify a failure and fix it without opening an editor locally.
Watch it explained
This is one section of Linux & the shell, 10 hours across 2 sections. Read it in full to get the whole topic in order.
Stop 40 of 50 on the AI Engineer route
0 of 50 topics done · 0h behind you, 603h to go
What you have marked done is stored in this browser only. It does not reach a server, and clearing site data will clear it.