HTTP & APIs
Requests, status codes, retries, rate limits.
When your program uses a service on another machine, it sends an HTTP request across the network and waits for a response. The request says what you want and carries your data; the response carries the result and a status code, a three-digit number reporting whether it worked. Calling a language model is exactly this and nothing more exotic.
Networks fail in specific, repeatable ways: requests time out, the provider tells you to slow down, connections drop halfway through a response. Code that does not handle those produces an intermittent bug your users hit and you cannot reproduce. Most reports that the model is flaky are unhandled rate limits or timeouts in the caller.
It sits directly under the Messages API and every provider SDK, which are thin wrappers over requests like these, and it is the level you drop to when one misbehaves. It pairs with async and concurrency, which is about having many of these calls in flight at once.
Anything crossing a network fails sometimes; your code decides whether that failure is handled or visible to a user.
In more detailCalling a model is an HTTP request, and everything that can go wrong with HTTP can go wrong with it: timeouts, 429s, truncated responses, connection resets. Idempotency, exponential backoff with jitter, and connection pooling are what separate a demo from something that survives real traffic.
Learn first
Requests, status codes and headers
By the end: You can read an API's documentation and call it correctly the first time.
- Methods and what each implies
- Status codes worth knowing by heart
- Headers that matter
- Reading an error response properly
Model providers are ordinary web APIs, so most of what goes wrong with them is ordinary HTTP going wrong. This section is the vocabulary you need to read a provider's documentation and get a call right first time: what actually goes in a request, what the status code coming back is telling you, and which headers carry the parts that matter. It is also how you work out from a failure whether the fault is your credentials, your payload or their servers.
The shape of a request#
A request is a method, a URL, some headers and usually a body. A response is a status code, some headers and a body. Everything else is detail.
import httpx
r = httpx.post(
"https://api.example.com/v1/messages",
headers={
"Authorization": f"Bearer {api_key}",
"Content-Type": "application/json",
},
json={"model": "small", "messages": [...]},
timeout=60,
)
r.status_code # 200
r.json() # parsed body
r.headers # rate limit info lives hereMethods, and what each one promises#
The method is a promise to the caller about what happens if the request is repeated, and that promise is what every proxy, retry layer and browser in the path relies on. Safe means the call does not change anything. Idempotent means sending it twice leaves the world in the same state as sending it once, which is not the same as getting the same response back.
method safe idempotent body typical use
GET yes yes no fetch a thing. Never use it to change one.
HEAD yes yes no the headers only: size, type, existence
POST no NO yes create, or "do this". Retrying may duplicate.
PUT no yes yes replace the whole resource at this URL
PATCH no no yes partial update. Idempotent only if you design it so.
DELETE no yes no remove it. Second call returns 404, state is the same.
OPTIONS yes yes no what this endpoint allows. CORS preflight uses it.Two consequences follow and both bite in production. A GET that changes state will eventually be repeated by something you do not control, such as a link prefetcher or a retrying proxy. And because POST is not idempotent, a retry after a timeout can charge you twice for the same generation. The fix is an idempotency key: a value you generate per logical operation and send in a header, which the server uses to recognise a repeat and return the original result instead of doing the work again.
Status codes to know cold#
code meaning whose fault what to do
200 fine - use the body
201 created something - read the Location header
204 done, no body - do not call .json() on it
206 partial content - you sent a Range header
301 moved permanently - update your URL, it is not coming back
304 not modified - your cached copy is still good
400 malformed request yours read the body, it names the field
401 not authenticated yours the key is missing, wrong or expired
403 authenticated, denied yours the key is real and lacks the permission
404 no such thing probably a wrong path far more often than missing data
405 wrong method yours you sent POST to a GET endpoint
409 conflict yours something changed under you; re-read and retry
413 payload too large yours your prompt or file exceeded their limit
415 unsupported media type yours you forgot Content-Type: application/json
422 understood, invalid yours valid JSON, impossible request
429 rate limited shared back off, read Retry-After
500 server error theirs retry with backoff
502 bad gateway theirs transient, retry
503 unavailable theirs overloaded or deploying, retry
504 gateway timeout theirs their upstream was slow. May have run anyway.4xx and 5xx need different handling
A 4xx means retrying the identical request will fail identically, so retrying is wasted time and money. A 5xx or a timeout is often transient and worth retrying. Code that retries everything is as wrong as code that retries nothing. The one exception is 429, which is a 4xx that you must retry, because the thing that needs to change is the time rather than the request.
A provider that misbehaves on demand#
Reading about status codes is not the same as having handled them, and you cannot make somebody else's API fail on request. Run one of your own instead. The server below is about forty lines of standard library, returns each of the failures you care about on demand, and gives you something to point a client at while you are still writing it.
import json, threading, urllib.error, urllib.request
from http.server import BaseHTTPRequestHandler, HTTPServer
class FakeProvider(BaseHTTPRequestHandler):
"""Behaves like a model provider on a bad day, so the client can be tested."""
def _send(self, status, payload, extra=None):
body = json.dumps(payload).encode()
self.send_response(status)
self.send_header("Content-Type", "application/json")
self.send_header("request-id", "req_0001")
for k, v in (extra or {}).items():
self.send_header(k, v)
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)
def do_POST(self):
n = int(self.headers.get("Content-Length", 0))
body = json.loads(self.rfile.read(n) or b"{}")
auth = self.headers.get("Authorization", "")
token = auth[7:] if auth.startswith("Bearer ") else ""
if self.path != "/v1/messages":
return self._send(404, {"error": {"type": "not_found", "message": "unknown path"}})
if not token:
return self._send(401, {"error": {"type": "authentication_error",
"message": "missing bearer token"}})
if token == "readonly":
return self._send(403, {"error": {"type": "permission_error",
"message": "key cannot write"}})
if "messages" not in body:
return self._send(400, {"error": {"type": "invalid_request_error",
"message": "messages: field required"}})
if body.get("model") == "busy":
return self._send(429, {"error": {"type": "rate_limit_error",
"message": "slow down"}},
{"Retry-After": "2",
"x-ratelimit-remaining-requests": "0",
"x-ratelimit-reset-requests": "2s"})
return self._send(200, {"id": "msg_01", "content": "hello"},
{"x-ratelimit-remaining-requests": "59"})
def log_message(self, *args):
pass # keep the test output clean
server = HTTPServer(("127.0.0.1", 0), FakeProvider)
threading.Thread(target=server.serve_forever, daemon=True).start()
BASE = "http://127.0.0.1:%d" % server.server_address[1]
def call(path="/v1/messages", token="sk-test", payload=None, timeout=5):
req = urllib.request.Request(
BASE + path,
data=json.dumps(payload if payload is not None else
{"model": "small", "messages": []}).encode(),
headers={"Authorization": "Bearer " + token,
"Content-Type": "application/json"},
method="POST",
)
try:
with urllib.request.urlopen(req, timeout=timeout) as r:
return r.status, dict(r.headers), json.loads(r.read())
except urllib.error.HTTPError as e:
return e.code, dict(e.headers), json.loads(e.read())
for label, kwargs in (
("ok", {}),
("no token", {"token": ""}),
("read-only key", {"token": "readonly"}),
("wrong path", {"path": "/v1/message"}),
("bad payload", {"payload": {"model": "small"}}),
("rate limited", {"payload": {"model": "busy", "messages": []}}),
):
status, headers, body = call(**kwargs)
detail = body.get("error", {}).get("message", body.get("content", ""))
print("%-14s %3d %-28s retry-after=%s"
% (label, status, detail, headers.get("Retry-After")))
# ok 200 hello retry-after=None
# no token 401 missing bearer token retry-after=None
# read-only key 403 key cannot write retry-after=None
# wrong path 404 unknown path retry-after=None
# bad payload 400 messages: field required retry-after=None
# rate limited 429 slow down retry-after=2One detail in that handler is worth copying rather than skimming. The 401 check tests the token, not the header: an empty Authorization header still starts with the word Bearer, so a naive prefix check accepts a request with no credentials in it. That is a real class of authentication bug, and writing the server is how you meet it.
Turning a response into a decision#
Callers do not want a status code, they want to know what to do next, and there are only four answers. Writing that mapping down once, in one function, is the difference between a client that degrades and one that produces a stack trace.
def triage(status, headers, body):
"""Turn a response into one of four actions. Everything a caller needs to
decide is in the status; everything a human needs to fix it is in the body."""
err = body.get("error", {})
where = err.get("message", "")
rid = headers.get("request-id", "-")
if status < 400:
return "use it", ""
if status in (408, 429) or status >= 500:
wait = headers.get("Retry-After")
return "retry", "after %s s" % wait if wait else "with backoff"
if status in (401, 403):
return "fix the credential", where
return "fix the request (id %s)" % rid, where
for label, kwargs in (
("ok", {}),
("no token", {"token": ""}),
("bad payload", {"payload": {"model": "small"}}),
("rate limited", {"payload": {"model": "busy", "messages": []}}),
):
action, detail = triage(*call(**kwargs))
print(("%-14s -> %-26s %s" % (label, action, detail)).rstrip())
# ok -> use it
# no token -> fix the credential missing bearer token
# bad payload -> fix the request (id req_0001) messages: field required
# rate limited -> retry after 2 sHeaders that matter#
on the way out
Authorization Bearer <key>. Some providers use their own header instead.
Content-Type application/json, or the upload fails with 415
Accept application/json, or text/event-stream for streaming
Idempotency-Key your id for this operation, so a retry cannot duplicate it
User-Agent name your service. It is what appears in their logs.
on the way back
request-id the one thing support will ask you for. Log it always.
Retry-After seconds, or a date. Obey it rather than guessing.
x-ratelimit-remaining-requests how many calls are left in this window
x-ratelimit-remaining-tokens the limit that surprises people
Content-Type check it before parsing. An HTML error page is not JSON.The last line of that list catches a failure that looks baffling. When something in front of the API fails, a load balancer or a proxy, you get an HTML error page with a 502 status, and code that calls json on every response raises a parse error instead of reporting the 502. Check the status first, then the content type, then parse.
Read the error body#
Most APIs put a useful message in the body of an error response, and most code throws it away and logs only the status. That habit turns a five-second fix into an afternoon.
if r.status_code >= 400:
log.error("api_error", extra={
"status": r.status_code,
"body": r.text[:500], # the part that says what went wrong
"request_id": r.headers.get("request-id"),
})Truncate rather than logging the whole body, because an error body can be large and a failing endpoint produces a lot of them. Five hundred characters reaches the message on every provider worth using. And never log the request body alongside it without thinking, since that is where the user's prompt lives.
Where people go wrong
Not setting a timeout. The default in many clients is to wait indefinitely, so one stalled connection hangs a process until someone notices.
Build this: Call an API and deliberately trigger a 400, a 401 and a 429. Log the full error body each time and note how much more useful it is than the status code alone.
Open this section on its own pageRetries, rate limits and timeouts
By the end: Your API client survives a bad afternoon on the provider's side.
- Exponential backoff with jitter
- Respecting rate limit headers
- Timeouts at every level
- Circuit breaking
Every call to a model provider will fail sooner or later, and most of those failures are temporary: a rate limit, an overloaded backend, a network blip. Handling them is not optional engineering polish, because a service that gives up on the first 429 is a service that is down whenever the provider is busy. This section covers retrying in a way that helps rather than hurts, reading the rate limit headers the provider already sends you, setting timeouts at every level, and stopping entirely when the thing you are calling is genuinely broken.
What is worth retrying#
status meaning retry?
408 request timeout yes
429 rate limited yes, and read Retry-After
500 server error yes
502/503 bad gateway / unavailable yes
504 gateway timeout yes
400 malformed request no. it will fail identically
401/403 auth no. fix the credential
404 not found no
422 semantically invalid noExponential backoff with jitter#
Backoff means waiting longer after each failure — one second, two, four, eight — so that a struggling service gets progressively more room. Jitter means randomising that wait. The second half is the one people leave out, and it is the half that matters when more than one client is involved, because without it every client that failed at the same moment retries at the same moment, forever.
import random
def simulate(jitter, clients=100, capacity=20, seed=0):
"""Every client fails at t=0 and retries. capacity is how many the server
can serve at one instant. Returns the size of each arrival burst."""
rng = random.Random(seed)
pending = [(0.0, 0) for _ in range(clients)] # (next attempt, attempt no.)
bursts = []
while pending:
t = min(p[0] for p in pending)
due = [p for p in pending if p[0] <= t + 1e-9]
bursts.append(len(due))
rest = due[min(len(due), capacity):] # the rest are rejected again
pending = [p for p in pending if p[0] > t + 1e-9]
for _, n in rest:
base = 2 ** n # 1, 2, 4, 8 seconds
pending.append((t + (rng.uniform(0, base) if jitter else base), n + 1))
if len(bursts) > 200:
break
return bursts, t
for jitter in (False, True):
bursts, done = simulate(jitter)
print(jitter, bursts[:6], "cleared at t =", round(done, 2))
# False [100, 80, 60, 40, 20] cleared at t = 15.0
# True [100, 1, 1, 1, 1, 1] cleared at t = 0.98That synchronised wave is the thundering herd, and the reason it matters more than it looks is that the clients caused the problem they are waiting for. Full jitter — a uniform random wait between zero and the backoff ceiling, which is what the code above does — is the simplest version and is hard to improve on. The production form of this adds two things: a cap on the delay so it does not grow to minutes, and a limit on attempts so a request cannot retry forever.
import time
RETRYABLE = {408, 429, 500, 502, 503, 504}
def retryable(err):
"""Could the identical request succeed if sent again?"""
return getattr(err, "status_code", None) in RETRYABLE
def with_retries(call, attempts=5, cap=30.0):
for n in range(attempts):
try:
return call()
except Exception as e:
if n == attempts - 1 or not retryable(e):
raise
delay = min(cap, 2 ** n)
time.sleep(random.uniform(0, delay)) # full jitter
raise RuntimeError("unreachable")Respecting rate limit headers#
Guessing the backoff is the fallback. Most providers tell you the answer directly, and a client that reads the headers is better behaved and faster than one that does not — it waits exactly as long as required rather than doubling blindly past it.
Retry-After seconds, or a date. Obey it exactly.
x-ratelimit-limit-requests your ceiling for this window
x-ratelimit-remaining-requests how many you have left
x-ratelimit-reset-requests when the window refills
x-ratelimit-remaining-tokens most providers limit tokens as well as callsThe remaining counters are the more useful half, because they let you avoid the error rather than recover from it. Read them on every response and slow down as the remaining count approaches zero, and a well-behaved client rarely sees a 429 at all. The token counter is the one that surprises people: you can be far inside the request limit and still be throttled because a few long prompts consumed the token budget.
Timeouts at every level#
A request with no timeout can hang until the operating system gives up, which may be minutes, and during that time it holds a connection, a worker and a user's attention. The mistake is setting one timeout and believing you are covered, when there are several and they nest.
connect timeout establishing the TCP connection ~5 s
read timeout waiting for the next byte model calls: 60 s+
total timeout the whole operation bounds everything
retry budget total time across all attempts what the caller feels
request deadline propagated from the incoming call the honest ceilingRetries multiply the timeout
A 30-second timeout with five retries is a two-and-a-half-minute worst case, and that is the number your caller experiences. The common failure is a service where each layer independently retries: three layers each retrying three times is twenty-seven calls for one request, and a provider blip becomes a self-inflicted outage. Decide which single layer owns retrying, and make the others fail fast.
Circuit breaking#
Retrying assumes the failure is temporary. When it is not — the provider is down, the credential has been revoked, the region is unreachable — retrying makes everything worse: each request waits through its full backoff before failing, workers pile up, and a dependency that is merely broken becomes an outage in your own service. A circuit breaker notices a run of failures and stops calling for a while.
closed calls pass through counting failures
open calls fail immediately no waiting, no load on the dependency
half-open one trial call is allowed success closes it, failure re-opensOpen it on a failure rate over a window rather than on a count of consecutive failures, or a steady trickle of successes will hold it closed while most requests fail. And make what happens in the open state a deliberate decision rather than an exception — that is exactly where the fallback chain from the model-selection topic belongs.
Where people go wrong
Retrying 4xx errors. A malformed request stays malformed, so five retries produce five identical failures, five times the latency and five times the log noise.
Build this: Write a client with backoff, jitter, rate limit awareness and a total timeout. Point it at a server you control that returns 429 and 500 on demand, and confirm the behaviour in each case.
Open this section on its own pageWhere you use this
A topic is easier to care about when you can see what it is for. These projects lean on it at a named stage.
- Stage 4, Make failure survivable: Timeouts, backoff and jitter for the tool that talks to something over the network.
- Stage 1, A model behind an endpoint: Status codes and timeouts, and what the endpoint should say when it is already busy.
Now actually do it
Reading a topic is not learning it. These are the hours that count, and each one tells you when you are finished.
1Build a resilient client against a hostile server
6hWrite a small server that returns 429, 500 and hangs on demand. Then a client with backoff, jitter, rate limit header awareness, per-request and total timeouts, and a circuit breaker.
Done when: Each failure mode behaves correctly, and the client never retries a 400.
2Idempotency under retry
2hAdd a state-changing endpoint and make it idempotent with a key. Simulate a timeout where the server actually succeeded, retry, and confirm the action happened exactly once.
Done when: The double-execution case is provably handled, with a test.
8 hours of practice
Interview questions
6 questions, grouped by the section that teaches the answer. An answer you have read is not an answer you can give, so use Test yourself and say it out loud first.
Requests, status codes and headers
What is the difference between 401 and 403?
401 means the request was not authenticated, so the credentials are missing or invalid. 403 means authentication succeeded but the caller is not permitted to do this. One is a credentials problem, the other a permissions problem.
Which HTTP errors should you retry?
429 after a backoff, and 5xx and network timeouts, which are often transient. Not 4xx generally, because an identical request will fail identically, so retrying wastes time and money.
What is idempotency and why does it matter for API calls?
An idempotent request has the same effect whether it runs once or several times. It matters because a timeout does not tell you whether the server acted, so a retry can duplicate the action. An idempotency key lets the server recognise the repeat and return the original result.
Retries, rate limits and timeouts
Why add jitter to exponential backoff?
Without it, all clients that failed at the same moment retry at the same moment, producing a synchronised spike that can knock over the recovering service again. Randomness spreads the retries out.
You are calling an API 1000 times and keep hitting rate limits. What do you do?
Cap concurrency with a semaphore rather than firing everything at once, read the rate limit headers and slow down before being rejected, back off exponentially with jitter and honour retry_after, and use batch endpoints for anything that can wait.
What does a timeout tell you about whether the server acted?
Nothing. The request may have been processed while the connection dropped. That is why any retry of a state-changing call needs an idempotency key, or you risk doing it twice.
Resources
Free to watch and read, picked for this topic.
For requests, status codes and headers
For retries, rate limits and timeouts
Full courses: the whole topic, start to finish
Stop 2 of 50 on the AI Engineer route
0 of 50 topics done · 0h behind you, 603h to go
What you have marked done is stored in this browser only. It does not reach a server, and clearing site data will clear it.