Deploying one AI service, step by step

You have a container that runs your service locally; this is the walk from there to a URL other people can call. The path is much the same on every cloud: push the image to a registry and point a managed container service at it. The work is in the four things that are easy to skip afterwards, which are the API key coming from a secret manager, an identity that can do only what this service needs, minimum and maximum instance counts, and a budget alert. Left at their defaults you get an admin role, a slow first request every morning, and no ceiling on the bill.

Pick the simplest thing that runs a container#

For most AI APIs, which are containers that call a model provider, a managed container service is enough: Google Cloud Run, AWS App Runner or ECS on Fargate, Azure Container Apps. You give it an image; it runs it, scales it and gives you a URL. Kubernetes is for when you have many services and a team to run it.

Shape of the work                    Run it on
HTTP API that calls a model          managed container service
Long job, minutes per item           queue plus a worker container
Scheduled batch, once a night        a scheduled job, not a running service
Needs a GPU at request time          GPU-backed service, minimum one instance
Many services, a platform team       Kubernetes, and not before

The tell that you have picked the wrong shape is a request timeout. A managed container service will happily hold a connection for a minute or so, but a job that takes five minutes does not belong behind an HTTP request no matter how generous the timeout is, because the user closed the tab and you are still paying for the work. Return an id, do the work in a queue worker, and let them poll or be notified.

The path#

build image  ->  push to registry  ->  service pulls it  ->  HTTPS URL
                     (Artifact Registry,      (Cloud Run, ECS,
                      ECR, ACR)                Container Apps)

Two details in that first arrow decide whether you can ever roll back. Build for the architecture the platform runs, which is usually linux/amd64 and is not what an Apple laptop builds by default; getting this wrong produces a container that starts and immediately dies with an exec format error. And tag with the commit hash, not only with latest, because latest is a moving name and a deploy that overwrites it has no previous version to return to.

SHA=$(git rev-parse --short HEAD)

docker build --platform linux/amd64 -t myapi:$SHA .
docker tag myapi:$SHA REGISTRY/myapi:$SHA
docker push REGISTRY/myapi:$SHA

# deploy the hash, never the moving tag
gcloud run deploy myapi --image REGISTRY/myapi:$SHA --region europe-west1

Secrets#

Store the model API key in the cloud's secret manager and let the service read it at startup as an environment variable. It never appears in the image, the repository or the deploy script, and rotating it does not require a rebuild.

The configuration references the secret by name and version. That reference is safe to commit, which matters more than it sounds: it means the deploy is reproducible from the repository, and the only thing a new engineer cannot recreate is the value itself, which is exactly right.

Identity#

Give the service its own identity (a service account or role) with permission to read that one secret and write logs, and nothing else. If the service is compromised, the attacker gets only what the identity can do.

Write the list before you write the policy. For a typical AI API it is four lines, and if yours is longer than eight the service is probably doing two jobs.

read   the one secret holding the model API key
write  logs
read   the models bucket, and list it
write  the outputs bucket
                       everything else: denied, including by omission

The deploy identity is a separate thing and people conflate them. The credentials in CI need to push an image and update a service; they do not need to read your production database. Two identities, two scopes, and CI's one is the more dangerous of the pair because it can replace the running code.

Scaling and cold starts#

• Services scale to zero when idle, which is cheap, but the first request after idle waits for a container to start.
• Set a minimum of one instance for anything user-facing, and keep the image small so starts are quick.
• Set a maximum so a traffic spike or a bug cannot run up an unbounded bill, and remember your model provider has rate limits too.
• Set the request timeout above your slowest model call, including streaming.

A cold start is the sum of three things: pulling the image if the node does not have it, starting the container, and your application's own import and initialisation before it answers anything. The last one is the part you control and the part people forget. Measured locally with a warm image, the same service takes 0.75 seconds to become ready when its module imports nothing heavy, and 1.7 seconds when it imports scikit-learn and constructs a model at import time.

src.main:app  ready in 0.79s      # a bare FastAPI app
src.heavy:app ready in 1.52s      # the same app, importing sklearn at module level
src.main:app  ready in 0.70s
src.heavy:app ready in 1.60s
src.main:app  ready in 0.77s
src.heavy:app ready in 1.90s

So the checklist for a slow first request, in the order worth checking. Is the minimum instance count zero. Is the image bigger than it needs to be. Is the application loading a model, opening a connection pool or reading a large file before it serves. The first two are settings; the third is code, and the fix is usually to make the expensive thing lazy and let the readiness check report not ready until it has finished.

Concurrency is the setting most specific to AI services and the one most often left wrong. A container serving an API that spends its time waiting on a model provider can handle many simultaneous requests on one CPU, because almost none of it is computation. The ceiling that matters is not your CPU, it is the provider's rate limit, and the number worth setting is maximum instances multiplied by per-instance concurrency staying under it. Otherwise scaling up turns one slow response into a wall of 429s.

Rolling back#

Managed services keep previous revisions, so a rollback is shifting traffic to the last good one. It takes seconds, needs no build and no repository, and works when the person doing it does not understand the change. That is the property that matters at two in the morning.

It only works if two things were true before the incident. Deploys are immutable images tagged by commit, so the previous revision still exists and still points at real bytes. And the database migration in that release was backwards compatible, because traffic can move back in seconds and a dropped column cannot. Add columns, deploy, backfill, and only remove the old one a release later.

Before you forget#

• A budget alert on the cloud account and a spending limit on the model provider account.
• Logs going somewhere searchable, with request ids.
• A health check endpoint the platform can call.

Add the two limits that actually stop things rather than reporting them: a maximum instance count on the service, and a hard spending cap with the model provider. A budget alert arrives after the money is gone, and for a runaway retry loop calling a frontier model that can be a day's spend before anybody reads the email.

What to skip#

• A custom domain and TLS certificates on the first deploy. The platform gives you an HTTPS URL; use it until something needs the pretty name.
• A staging environment identical to production. A separate project with the same deploy script and smaller instances is what people actually maintain.
• Autoscaling policies tuned by hand. Set a minimum, a maximum and a concurrency number, and leave the rest alone.
• Blue-green and canary machinery for one service with no traffic. Revisions and a one-command rollback cover it.