Cloud fundamentals

Cloud platforms have hundreds of services and you need about a dozen. This section is that subset: run a container somewhere, put objects in a bucket, give the service an identity with exactly the access it needs, get secrets to it without committing them, and see what it is costing you before the invoice does. Permissions are the part that bites, in both directions — too little and nothing works, too much and one compromised component can reach everything you own.

Six primitives, three vocabularies#

Almost every system you will be asked to design is built from six pieces. The pieces are the same on every cloud; only the names change, and being lost in the names is what makes cloud feel enormous when it is not. Learn the row, not the column.

What it is           AWS              Google Cloud        Azure
Object storage       S3               Cloud Storage       Blob Storage
Run a container      App Runner,      Cloud Run           Container Apps
                     ECS on Fargate
A whole machine      EC2              Compute Engine      Virtual Machines
Managed SQL          RDS              Cloud SQL           Azure Database
A queue              SQS              Pub/Sub             Service Bus
Identity             IAM role         service account     managed identity
Secrets              Secrets Manager  Secret Manager      Key Vault
Logs and metrics     CloudWatch       Cloud Logging       Azure Monitor
Image registry       ECR              Artifact Registry   Container Registry

What each one is actually for, in the order you will meet them.

Object storage is a very large dictionary: you put bytes under a key and get them back by that key. It is not a filesystem. There are no real directories, only keys with slashes in them, you cannot append to an object or edit part of one, and listing is slow and paginated. It is where datasets, model weights, embeddings dumps and anything else large and immutable belong, at roughly two cents per gigabyte per month. If you are reaching for it to store something you will update frequently, you want a database.

A container runtime is the thing that takes the image you built and turns it into a URL. The managed kind scales it up when requests arrive, scales it to zero when they stop, and charges you for the time it was running. This is where nearly every AI service should live, because an AI API spends most of its life waiting on a model provider rather than using CPU.

A queue decouples two services so the slow one cannot take down the fast one. A request arrives, you write a message and return immediately; a worker reads it and does the ten seconds of model calls. It also gives you retries and a dead letter queue for the messages that failed repeatedly, which is where the interesting bugs collect. The moment any request in your system takes longer than a user will wait, a queue is the answer and everything else is a workaround.

A managed database is Postgres that somebody else backs up, patches and fails over. You give up the ability to install any extension you like and get back the hours you would have spent on replication. For anything with users in it, take the managed one.

Identity is the subject of the next heading, and is the only one of the six that is genuinely conceptual rather than a product.

Logs are where you will be at two in the morning, and the only thing that makes them useful is a request id attached to every line so you can follow one request through four services. Structured logs, one JSON object per line, are searchable; a formatted sentence is not. Set retention explicitly on the first day, because the default is often forever and it is billed by the gigabyte.

Identity and least privilege#

Every service gets an identity, and that identity gets permissions. The rule is to grant the minimum: this service reads from this bucket, and nothing else.

IAM looks like a configuration format, which is why people learn its syntax and still cannot predict what it will do. It is better understood as a function. A request arrives carrying an identity, an action and a resource, and the platform returns allow or deny by the same three steps everywhere: an explicit deny anywhere wins immediately, otherwise an explicit allow wins, otherwise the answer is deny. Nothing is permitted by default, and no allow can override a deny.

import fnmatch

POLICY = [
    {"effect": "Allow", "action": "s3:GetObject",  "resource": "arn:aws:s3:::models/*"},
    {"effect": "Allow", "action": "s3:PutObject",  "resource": "arn:aws:s3:::outputs/*"},
    {"effect": "Allow", "action": "s3:*",          "resource": "arn:aws:s3:::scratch/*"},
    {"effect": "Deny",  "action": "s3:DeleteObject", "resource": "*"},
]

def decide(policy, action, resource):
    """The three-step rule every cloud uses. Order is the whole point."""
    matches = [
        s for s in policy
        if fnmatch.fnmatch(action, s["action"]) and fnmatch.fnmatch(resource, s["resource"])
    ]
    if any(s["effect"] == "Deny" for s in matches):
        return "DENY (explicit deny wins)"
    if any(s["effect"] == "Allow" for s in matches):
        return "ALLOW"
    return "DENY (nothing allowed it)"

s3:GetObject     arn:aws:s3:::models/v3/weights.bin     -> ALLOW
s3:GetObject     arn:aws:s3:::payroll/2026.csv          -> DENY (nothing allowed it)
s3:PutObject     arn:aws:s3:::outputs/run-12.json       -> ALLOW
s3:PutObject     arn:aws:s3:::models/v3/weights.bin     -> DENY (nothing allowed it)
s3:DeleteObject  arn:aws:s3:::scratch/tmp.bin           -> DENY (explicit deny wins)
s3:ListBucket    arn:aws:s3:::models                    -> DENY (nothing allowed it)

That last line is the mistake everyone makes once. Reading an object and listing a bucket are different actions on different resources: the objects are models slash star, but the bucket itself is just models, with no slash and no star. So a policy that lets the service read every file in the bucket still cannot list what is in it, and the failure arrives as a confusing access denied on a call you were sure you had permitted. When a permission fails and you are certain it should not have, check whether the action operates on the container rather than the contents.

The practical method, in order. Start with nothing. Run the thing. Read the denial, which names the exact action and resource. Add that one permission. Repeat until it works. It takes fifteen minutes and produces a policy that is correct by construction, where starting from a wildcard and narrowing it down is a task nobody ever finishes.

Two more rules matter more than the syntax. Give the identity to the workload, not to a person: a service should get a role attached to it by the platform and receive short-lived credentials automatically, so there is no long-lived key anywhere to leak. And use a separate account or project per environment. Scoping by naming convention inside one account fails the first time somebody typos a resource name, and then staging is writing to production.

Secrets#

• A secret manager, injected as environment variables at runtime.
• Never in the image, never in the repository, never in a log line.
• Rotate them, and make sure rotation does not require a code change.
• Scope keys per environment, so a leaked staging key cannot touch production.

Reference the secret by name in the deploy configuration and let the platform fetch the value at start-up. The name is safe to commit; the value never appears in your repository, your image or your CI logs. Rotating then means writing a new version into the secret manager and restarting, with no rebuild and no code change, which is the only version of rotation anyone actually does twice.

What a bill is actually made of#

Cloud bills surprise people because the expensive thing is rarely the obvious one. For AI workloads it is usually idle GPU instances, data transfer between regions, and logs retained forever.

It is worth seeing the arithmetic once. Below is a month for a small team, at illustrative rates in roughly the right order of magnitude. Check the current prices before quoting a number to anyone; the shape is the part that does not change.

GPU box, forgotten after Friday's experiment                   2,190.00
NAT gateway, 3 TB through it                                     135.00
Logs: 200 GB ingested, every request body logged                 100.00
Logs: 2.4 TB retained forever, month 12                           72.00
Cross-AZ chatter: app and database in different zones, 3 TB       60.00
Object storage, 2 TB of datasets                                  46.00
Egress: 500 GB of model responses to users                        45.00
The API container itself: 1 vCPU, 2 GB, always on                 43.80
NAT gateway, idle                                                 32.85
-----------------------------------------------------------------------
total                                                          2,724.65

Three line items surprise people, every time. The first is idle compute, and specifically a GPU: one mid-range card left running costs about the same over a month whether it trains anything or not, and it dwarfs everything else on the bill. Anything you start by hand needs an alarm or a schedule that stops it.

The second is data movement, which nobody budgets for because it does not look like a resource. Data arriving is free; data leaving costs around nine cents a gigabyte, and data crossing between availability zones inside the same region costs a cent a gigabyte in each direction, which is why an application in one zone chatting to a database in another can invent a bill from nothing. A NAT gateway, the box that lets private machines reach the internet, charges both by the hour and by the gigabyte through it, so a training job pulling a dataset pays twice.

The third is logs. They are billed on the way in, by the gigabyte ingested, and then again for storage every month you keep them. A service that logs the full request and response body at a few hundred requests a second generates more billable data than the application generates value, and because retention often defaults to never expire, month twelve pays for months one through eleven all over again.

• Set a budget alert on day one, before you have anything running.
• Tag resources by project, or you cannot attribute the bill later.
• Shut down development GPU instances. An idle one costs the same as a busy one.
• Set log retention deliberately. Indefinite retention is a slow, invisible expense.

Budget alerts do not stop spending; they tell you afterwards, and the smallest useful granularity is usually a day. The things that actually stop spending are a hard maximum instance count on the service, a spending cap on the model provider account, and an auto-stop schedule on anything with a GPU in it. Set those as well as the alert.

When managed is worth it, and when it is not#

The managed version of anything costs more per unit and less in hours. The question is never which is cheaper on the invoice; it is whether the hours you get back are worth more than the premium, and for a small team they almost always are. The exceptions are real but narrow.

Take the managed one when        Run it yourself when
it holds data you cannot lose    it is stateless and trivial to replace
failure means a page at 3am      a restart fixes it and nobody notices
it needs backups or failover     losing it costs one rerun
nobody on the team wants to      someone's actual job is running it
  own it                         the managed bill is most of your bill
you have fewer than ten people   you need a version or extension they refuse

The one that flips with scale is GPU compute. Renting on demand is right while your usage is spiky and small, because you pay only for the hours you use. Once a GPU is busy most of the day, a committed one-year rate or a reservation is substantially cheaper, and once you are running many of them continuously the arithmetic starts to favour owning hardware. The mistake is not choosing wrong at the start; it is never revisiting a decision made when the workload was a tenth of its current size.

Kubernetes is the same judgement in a different costume. It is the right answer for a dozen services and a platform team, and the wrong one for a single API, where it adds a control plane, a networking model and an ongoing upgrade obligation in exchange for capabilities you are not using. Start on a managed container service and move when the thing that hurts is actually orchestration.

What to skip#

• Certification. It examines breadth you will never use and none of the judgement above.
• Designing a VPC. Use the default network until something forces you off it, and even then use the platform's template.
• Learning a second cloud. The concepts transfer; the names are lookup.
• Infrastructure as code on day one for a single service. Get one thing deployed by hand first so you know what the code is describing, then write it down.
• Multi-region anything. It multiplies cost and failure modes, and almost nobody asking for it has an availability target that requires it.

You have a container that runs your service locally; this is the walk from there to a URL other people can call. The path is much the same on every cloud: push the image to a registry and point a managed container service at it. The work is in the four things that are easy to skip afterwards, which are the API key coming from a secret manager, an identity that can do only what this service needs, minimum and maximum instance counts, and a budget alert. Left at their defaults you get an admin role, a slow first request every morning, and no ceiling on the bill.

Pick the simplest thing that runs a container#

For most AI APIs, which are containers that call a model provider, a managed container service is enough: Google Cloud Run, AWS App Runner or ECS on Fargate, Azure Container Apps. You give it an image; it runs it, scales it and gives you a URL. Kubernetes is for when you have many services and a team to run it.

Shape of the work                    Run it on
HTTP API that calls a model          managed container service
Long job, minutes per item           queue plus a worker container
Scheduled batch, once a night        a scheduled job, not a running service
Needs a GPU at request time          GPU-backed service, minimum one instance
Many services, a platform team       Kubernetes, and not before

The tell that you have picked the wrong shape is a request timeout. A managed container service will happily hold a connection for a minute or so, but a job that takes five minutes does not belong behind an HTTP request no matter how generous the timeout is, because the user closed the tab and you are still paying for the work. Return an id, do the work in a queue worker, and let them poll or be notified.

The path#

build image  ->  push to registry  ->  service pulls it  ->  HTTPS URL
                     (Artifact Registry,      (Cloud Run, ECS,
                      ECR, ACR)                Container Apps)

Two details in that first arrow decide whether you can ever roll back. Build for the architecture the platform runs, which is usually linux/amd64 and is not what an Apple laptop builds by default; getting this wrong produces a container that starts and immediately dies with an exec format error. And tag with the commit hash, not only with latest, because latest is a moving name and a deploy that overwrites it has no previous version to return to.

SHA=$(git rev-parse --short HEAD)

docker build --platform linux/amd64 -t myapi:$SHA .
docker tag myapi:$SHA REGISTRY/myapi:$SHA
docker push REGISTRY/myapi:$SHA

# deploy the hash, never the moving tag
gcloud run deploy myapi --image REGISTRY/myapi:$SHA --region europe-west1

Secrets#

Store the model API key in the cloud's secret manager and let the service read it at startup as an environment variable. It never appears in the image, the repository or the deploy script, and rotating it does not require a rebuild.

The configuration references the secret by name and version. That reference is safe to commit, which matters more than it sounds: it means the deploy is reproducible from the repository, and the only thing a new engineer cannot recreate is the value itself, which is exactly right.

Identity#

Give the service its own identity (a service account or role) with permission to read that one secret and write logs, and nothing else. If the service is compromised, the attacker gets only what the identity can do.

Write the list before you write the policy. For a typical AI API it is four lines, and if yours is longer than eight the service is probably doing two jobs.

read   the one secret holding the model API key
write  logs
read   the models bucket, and list it
write  the outputs bucket
                       everything else: denied, including by omission

The deploy identity is a separate thing and people conflate them. The credentials in CI need to push an image and update a service; they do not need to read your production database. Two identities, two scopes, and CI's one is the more dangerous of the pair because it can replace the running code.

Scaling and cold starts#

• Services scale to zero when idle, which is cheap, but the first request after idle waits for a container to start.
• Set a minimum of one instance for anything user-facing, and keep the image small so starts are quick.
• Set a maximum so a traffic spike or a bug cannot run up an unbounded bill, and remember your model provider has rate limits too.
• Set the request timeout above your slowest model call, including streaming.

A cold start is the sum of three things: pulling the image if the node does not have it, starting the container, and your application's own import and initialisation before it answers anything. The last one is the part you control and the part people forget. Measured locally with a warm image, the same service takes 0.75 seconds to become ready when its module imports nothing heavy, and 1.7 seconds when it imports scikit-learn and constructs a model at import time.

src.main:app  ready in 0.79s      # a bare FastAPI app
src.heavy:app ready in 1.52s      # the same app, importing sklearn at module level
src.main:app  ready in 0.70s
src.heavy:app ready in 1.60s
src.main:app  ready in 0.77s
src.heavy:app ready in 1.90s

So the checklist for a slow first request, in the order worth checking. Is the minimum instance count zero. Is the image bigger than it needs to be. Is the application loading a model, opening a connection pool or reading a large file before it serves. The first two are settings; the third is code, and the fix is usually to make the expensive thing lazy and let the readiness check report not ready until it has finished.

Concurrency is the setting most specific to AI services and the one most often left wrong. A container serving an API that spends its time waiting on a model provider can handle many simultaneous requests on one CPU, because almost none of it is computation. The ceiling that matters is not your CPU, it is the provider's rate limit, and the number worth setting is maximum instances multiplied by per-instance concurrency staying under it. Otherwise scaling up turns one slow response into a wall of 429s.

Rolling back#

Managed services keep previous revisions, so a rollback is shifting traffic to the last good one. It takes seconds, needs no build and no repository, and works when the person doing it does not understand the change. That is the property that matters at two in the morning.

It only works if two things were true before the incident. Deploys are immutable images tagged by commit, so the previous revision still exists and still points at real bytes. And the database migration in that release was backwards compatible, because traffic can move back in seconds and a dropped column cannot. Add columns, deploy, backfill, and only remove the old one a release later.

Before you forget#

• A budget alert on the cloud account and a spending limit on the model provider account.
• Logs going somewhere searchable, with request ids.
• A health check endpoint the platform can call.

Add the two limits that actually stop things rather than reporting them: a maximum instance count on the service, and a hard spending cap with the model provider. A budget alert arrives after the money is gone, and for a runaway retry loop calling a frontier model that can be a day's spend before anybody reads the email.

What to skip#

• A custom domain and TLS certificates on the first deploy. The platform gives you an HTTPS URL; use it until something needs the pretty name.
• A staging environment identical to production. A separate project with the same deploy script and smaller instances is what people actually maintain.
• Autoscaling policies tuned by hand. Set a minimum, a maximum and a concurrency number, and leave the rest alone.
• Blue-green and canary machinery for one service with no traffic. Revisions and a one-command rollback cover it.