Containers

You can run your service on your laptop. This is how you make it run the same way somewhere else. A container bundles your code with its dependencies and enough of an operating system to behave identically in CI and in production. The section covers writing a Dockerfile that rebuilds in seconds rather than minutes, driving behaviour from environment variables so one image serves every environment, and the health checks an orchestrator reads to decide whether to send you traffic or restart you.

What a container actually is#

A container is not a small virtual machine. There is no second kernel and no emulated hardware. It is an ordinary Linux process with three things replaced: its view of the process table, its view of the filesystem, and its network interface. The kernel is the host's kernel. That is why a container starts in milliseconds where a virtual machine takes half a minute.

$ docker run --rm demo sh -c 'echo "my pid: $$"; ls /proc | grep -E "^[0-9]+$"'
my pid: 1
1 7 8 9

# Your process is PID 1 and can see four processes. The host has hundreds.
$ docker run --rm demo sh -c 'touch /tmp/mine; ls /tmp'
mine
$ docker run --rm demo sh -c 'ls /tmp'
                          # empty: the first container never existed here

An image and a container are different things and the words get used interchangeably to everyone's confusion. An image is a stack of read-only filesystem layers plus some metadata saying what to run. A container is one image, plus a thin writable layer on top, plus those isolation settings. Removing a container throws away the writable layer, which is why anything you edited inside a running container is gone the moment it stops. That is a feature: it is what makes an image the same everywhere.

What a layer is, and why order changes rebuild time#

Each instruction in a Dockerfile produces one layer, which is the set of file changes that instruction made. Docker caches layers and reuses one whenever the instruction and everything it depends on are unchanged. The catch is that a change to any layer invalidates every layer after it, because those were built on top of it. So the order of a Dockerfile decides how much of every rebuild you pay for.

# The slow one: the code is copied before the install, so any edit to any
# file invalidates the install layer.
FROM python:3.12-slim
WORKDIR /app
COPY . .
RUN pip install --no-cache-dir -r requirements.txt
CMD ["uvicorn", "src.main:app", "--host", "0.0.0.0", "--port", "8000"]

# The fast one: the dependency list is copied on its own and installed
# before any source code is anywhere near the image.
FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY src/ ./src/
CMD ["uvicorn", "src.main:app", "--host", "0.0.0.0", "--port", "8000"]

Change one character in one Python file and rebuild both. These are real timings from the two Dockerfiles above, on the same machine, with a warm cache.

$ time docker build -f Dockerfile.slow -t demo .
18.359 total
$ time docker build -f Dockerfile.fast -t demo .
0.225 total

The build output tells you which layers were reused, and it is worth learning to read. Anything marked CACHED cost nothing. The first line that is not CACHED is where your cache broke, and every line below it was rebuilt.

$ docker build --progress=plain -f Dockerfile.fast -t demo .
#6 [2/5] WORKDIR /app                                   CACHED
#7 [3/5] COPY requirements.txt .                        CACHED
#8 [4/5] RUN pip install --no-cache-dir -r requirements.txt   CACHED
#9 [5/5] COPY src/ ./src/                               DONE 0.0s

$ docker build --progress=plain -f Dockerfile.slow -t demo .
#6 [2/4] WORKDIR /app                                   CACHED
#7 [3/4] COPY . .                                       DONE 0.0s
#8 [4/4] RUN pip install --no-cache-dir -r requirements.txt   DONE 17.1s

Looking at the layer sizes makes the waste obvious. The layer being rebuilt on every commit holds 376 megabytes of installed packages. The layer that actually changed holds 140 bytes.

$ docker history demo --format '{{.Size}}\t{{.CreatedBy}}'
0B      CMD ["uvicorn" "src.main:app" ...
140B    COPY src/ ./src/
376MB   RUN pip install --no-cache-dir -r requirements.txt
80B     COPY requirements.txt .
0B      WORKDIR /app

The commands that matter, and what they say when it goes wrong#

docker build -t name .      build an image from the Dockerfile here
docker run -d -p 8000:8000 name   start it, detached, port mapped
docker ps                   what is running now
docker ps -a                what ran, including what died
docker logs NAME            its stdout and stderr, which is where the reason is
docker exec -it NAME sh     a shell inside a running container
docker stats --no-stream    live CPU and memory per container
docker compose up -d        start this directory's services together

The single most common confusion is a container that seems not to start. It did start; it started and then exited, so docker ps, which only lists running containers, shows nothing at all. The ladder is always the same three rungs: ps, then ps -a, then logs.

$ docker ps
                                        # nothing. Panic.
$ docker ps -a
q9dead    Exited (1) 2 seconds ago      # it ran and died
$ docker logs q9dead
  File "/app/src/main.py", line 6, in <module>
    API_KEY = os.environ["API_KEY"]
KeyError: 'API_KEY'

Exit codes are worth recognising on sight. Exit 1 is your program raising. Exit 137 is the kernel killing the process for using more memory than the container was allowed, which looks nothing like a crash from inside the application: there is no traceback, the log just stops mid-sentence.

$ docker run --memory=256m demo python -c "import numpy; ..."
allocated 448 MB
allocated 512 MB
                                  # no traceback, no error, just gone
$ docker inspect NAME --format 'ExitCode={{.State.ExitCode}} OOMKilled={{.State.OOMKilled}}'
ExitCode=137 OOMKilled=true

The other reliable time sink is the port. The flag is host port first, container port second, and there are two separate ways it fails. Either the host port is already taken, which says so clearly, or the process inside is listening on 127.0.0.1, which means the loopback interface inside the container's own network namespace. Nothing from outside can ever reach it, and the application's own startup log looks perfectly healthy.

$ docker run -p 8123:8000 demo
docker: Error response from daemon: ... Bind for 0.0.0.0:8123 failed:
port is already allocated

$ docker run -d -p 8124:8000 demo uvicorn src.main:app --host 127.0.0.1
$ docker logs NAME
INFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
$ curl localhost:8124/healthz
curl: (56) Recv failure: Connection reset by peer

Configuration#

The same image should run in every environment, with behaviour driven by environment variables. Baking configuration into the image means building a separate image per environment, which defeats the purpose.

import os

MODEL = os.environ.get("MODEL", "mid-tier")
MAX_CONCURRENCY = int(os.environ.get("MAX_CONCURRENCY", "8"))
API_KEY = os.environ["API_KEY"]        # no default: fail loudly if absent

Read every variable once, at import time, and let the process die if a required one is missing. The alternative, reading os.environ deep inside a request handler, turns a misconfiguration into an intermittent 500 an hour after the deploy went green.

Health checks#

Orchestrators need to know whether your service is alive and whether it is ready for traffic. These are different questions and conflating them causes restarts during startup.

@app.get("/healthz")      # alive: is the process functioning
def health():
    return {"ok": True}

@app.get("/readyz")       # ready: can it serve traffic yet
def ready():
    return {"ok": model_loaded and db.reachable()}

Liveness failing means restart me. Readiness failing means stop sending me requests, but leave me alone. An AI service makes the distinction expensive to get wrong, because loading a model into memory can take a minute or more. Point liveness at a single endpoint that returns a constant and touches nothing, and put every dependency check in readiness.

Symptom                                   Almost always
Restart loop, never finishes starting     liveness check waits on model load
Requests 502 for a minute after deploy    no readiness check at all
Healthy but every request times out       liveness hits a cached value, not the path
One bad database means a full restart     database check wired into liveness

Rolling out#

• Deploy the new version alongside the old and shift traffic gradually.
• Watch error rate and latency during the shift, not after.
• Keep rollback to one command. A rollback that needs a build is not a rollback.
• AI services need generous timeouts and concurrency limits driven by upstream rate limits rather than CPU.

Rollback only works if deploys are immutable. Tag every image with the commit hash as well as latest, and deploy the hash. Then rolling back is pointing the service at the previous hash, which takes seconds and needs no build, no network and nobody who understands the code. A deploy that overwrites the latest tag in place has no previous version to go back to.

What to skip#

• Kubernetes, until you have several services and someone whose job is to run it. A managed container service does the same job for one service with a tenth of the concepts.
• Writing your own base images. Use the official language image and move on.
• Docker networking beyond the default bridge and compose's service names. Custom networks, overlays and macvlan are for problems you do not have.
• Swarm. It exists, it works, nobody is hiring for it.
• Squeezing the last fifty megabytes out of an image. Getting from 1.5 gigabytes to 500 megabytes matters; getting from 500 to 450 does not.

A Dockerfile is a recipe, and the order of its lines decides both how long every rebuild takes and how much damage a compromised container can do. This section writes one for a Python AI service around the two things that matter most: installing dependencies before copying the code, so a one-line change rebuilds in seconds rather than minutes, and creating a user so the process is not root. It also covers what must never go in, which is secrets, model weights and your .git directory.

A good starting Dockerfile#

FROM python:3.12-slim AS base
WORKDIR /app

# Dependencies first: this layer is reused until requirements change
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Code last: changes here do not reinstall dependencies
COPY src/ ./src/

RUN useradd --create-home app
USER app

EXPOSE 8000
CMD ["uvicorn", "src.main:app", "--host", "0.0.0.0", "--port", "8000"]

Order is speed#

Each instruction makes a layer, and Docker reuses a layer if nothing it depends on changed. Copy the dependency list and install first, then copy the code. Editing a line of code then rebuilds in seconds instead of reinstalling everything. Measured on a service with fastapi, numpy, pandas and scikit-learn, the same one-character edit costs 18.4 seconds with the code copied first and 0.2 seconds with it copied last.

There are a few other ordering rules worth knowing, all of them the same idea applied in different places.

# Combine apt lines: two RUNs make two layers, and the second cannot
# remove files the first put in the layer below it.
RUN apt-get update && apt-get install -y --no-install-recommends git \
    && rm -rf /var/lib/apt/lists/*

# Pin versions. An unpinned install makes the cached layer a lie: it was
# built in March and you think it holds what pip would give you today.
COPY requirements.lock .
RUN pip install --no-cache-dir -r requirements.lock

# Put the line most likely to change as far down the file as it will go.
COPY src/ ./src/

Build in one stage, run in another#

A multi-stage build compiles or installs in a full image with build tools, then copies only the results into a slim final image. Compilers, headers and package caches are needed to install and useless afterwards, so they are dead weight in every pull and an extra toolkit for anyone who gets in.

FROM python:3.12 AS build
RUN apt-get update && apt-get install -y build-essential
RUN python -m venv /opt/venv
ENV PATH=/opt/venv/bin:$PATH
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

FROM python:3.12-slim
COPY --from=build /opt/venv /opt/venv
ENV PATH=/opt/venv/bin:$PATH
WORKDIR /app
COPY src/ ./src/
RUN useradd --create-home --uid 10001 app
USER app
CMD ["uvicorn", "src.main:app", "--host", "0.0.0.0", "--port", "8000"]

What the second stage removes is the compiler toolchain and the full Python base image underneath it. The numbers below are the same application built both ways on the same machine.

$ docker images demo --format '{{.Tag}}\t{{.Size}}'
single    1.52GB      # python:3.12 plus build-essential, kept
multi     532MB       # python:3.12-slim plus the venv

Do not spend the afternoon chasing the last fifty megabytes. The distance from 1.5 gigabytes to 500 megabytes is worth an hour; the distance from 500 to 450 is not. Alpine is the common wrong turn at this point. It is a genuinely tiny base image, but it uses the musl C library rather than glibc, and a project only ships a wheel for musl if it chose to. numpy and pandas now do. scikit-learn does not, so pip falls back to the source tarball and tries to compile it, in an image with no compiler.

$ docker build -f Dockerfile.alpine .
  Downloading numpy-2.2.1-cp312-cp312-musllinux_1_2_aarch64.whl      # fine
  Downloading pandas-2.2.3-cp312-cp312-musllinux_1_2_aarch64.whl     # fine
  Downloading scikit_learn-1.6.0.tar.gz (7.1 MB)                     # not fine
  Preparing metadata (pyproject.toml): finished with status 'error'
  ../meson.build:1:0: ERROR: Unknown compiler(s): [['cc'], ['gcc'], ...]
ERROR: failed to solve: pip install ... did not complete successfully: exit code: 1

Keep things out#

# .dockerignore
.git
.env
.venv
__pycache__
data/
*.ipynb

The build context is everything in the directory, and it is sent to the daemon before a single instruction runs. Without a .dockerignore, a repository with a checkout history and a local dataset in it sends all of that over, every build, and a COPY . . then bakes it into the image.

# no .dockerignore, with a 60 MB parquet file and a .git directory present
#6 transferring context: 80.02MB 1.4s done
$ docker images demo --format '{{.Size}}'
609MB

# same repository, .dockerignore added
#6 transferring context: 123B 0.0s done
$ docker images demo --format '{{.Size}}'
529MB

Do not run as root#

By default a container runs as root. It is easy to prove and easy to forget.

$ docker run --rm demo id
uid=0(root) gid=0(root) groups=0(root)

# after adding: RUN useradd --create-home --uid 10001 app  /  USER app
$ docker run --rm demo id
uid=10001(app) gid=10001(app) groups=10001(app)

Root inside a container is not root on the host, so this is not a catastrophe on its own. What it costs is every layer of defence after the first. A remote code execution in your service becomes root in the container, which can install tooling, read every file in the image including any configuration you thought was internal, and write as root into any mounted volume. That last one bites teams in normal operation, not just under attack: a container that writes its output into a mounted directory leaves files owned by uid 0, and the next process, or the developer on the host, cannot delete them.

# as root
$ docker run --rm -v "$PWD/out:/data" demo sh -c 'touch /data/x; ls -ln /data'
-rw-r--r-- 1 0 0 0 Sep 13 13:28 x        # owned by uid 0

# as the app user, the obvious damage simply fails
$ docker run --rm demo sh -c 'touch /etc/evil'
touch: cannot touch '/etc/evil': Permission denied
$ docker run --rm demo sh -c 'apt-get install -y curl'
E: Could not open lock file /var/lib/dpkg/lock-frontend (13: Permission denied)

Two things break when you add USER and catch people out. Anything the process writes at runtime must be in a directory the user owns, so chown it in the Dockerfile. And a port below 1024 needs root to bind, which is why containers listen on 8000 or 8080 and the platform maps 443 to it from outside.

GPU images are big#

Images with CUDA and PyTorch are often several gigabytes. Start from an official base image that matches your CUDA version, keep model weights out of the image and download or mount them at startup, and expect slow first pulls on new machines.

Two specifics save real time here. Install the wheel that matches your hardware rather than the default: the plain pip install of torch pulls the full CUDA runtime, and on a CPU-only serving box the CPU wheel is a fraction of the size. And treat weights as data, not code. A ten gigabyte checkpoint in the image means every deploy transfers ten gigabytes and changing the model needs a rebuild, where fetching it at start-up from object storage means changing an environment variable.

# Serving on CPU: the index URL is the whole optimisation.
RUN pip install --no-cache-dir torch --index-url https://download.pytorch.org/whl/cpu

# Weights come from outside, chosen at run time.
ENV MODEL_URI=s3://models/classifier/v7
CMD ["python", "-m", "src.serve"]

Compose for local development#

services:
  api:
    build: .
    ports: ["8000:8000"]
    env_file: .env
    depends_on:
      db:
        condition: service_healthy
  db:
    image: pgvector/pgvector:pg16
    environment:
      POSTGRES_PASSWORD: local-only
    ports: ["5432:5432"]
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 2s
      retries: 15

Two things in that file are the whole reason to use compose rather than two terminal windows. The first is that services reach each other by service name on a private network: the api container connects to db on port 5432, not to localhost, because localhost inside a container is that container. Getting this wrong produces a connection refused that looks like the database is down when it is running perfectly.

$ docker compose exec api python -c "import socket; print(socket.gethostbyname('db'))"
172.19.0.2
$ docker compose exec api python -c "import socket; socket.create_connection(('localhost',5432),2)"
ConnectionRefusedError: [Errno 111] Connection refused

The second is the healthcheck condition. A plain depends_on waits for the container to have started, which for Postgres is several seconds before it accepts connections, so the api comes up, fails to connect and exits. Adding condition service_healthy makes compose wait for the check to pass, and the difference shows up in the start-up log.

# depends_on: [db]
Container db-1   Started
Container api-1  Starting        <- immediately, db not accepting yet

# depends_on: db: condition: service_healthy
Container db-1   Started
Container db-1   Waiting
Container db-1   Healthy
Container api-1  Starting        <- only now

docker compose up -d --build    rebuild changed images and start
docker compose logs -f api      follow one service
docker compose exec api sh      shell into a running service
docker compose down -v          stop and delete the volumes too

What to skip#

• Alpine for Python. It makes builds slower, not images smaller, once wheels have to be compiled.
• Distroless and scratch images, until something is actually scanning you. They complicate debugging because there is no shell to exec into.
• ONBUILD, and clever base images your team maintains. Repeat the six lines.
• Hand-tuning below about 500 megabytes for a Python service. The wins are gone.