Streaming responses

A model generates one token at a time, so the first word exists long before the last one. Streaming sends each piece as it is produced rather than waiting for the whole response, which changes a twenty-second wait into a response that starts in half a second. Nothing about the total time improves — the change is entirely in what the user experiences — and it costs you a harder failure model, because by the time something goes wrong you have already sent a success. This section covers consuming a stream correctly, what it breaks, and how errors arrive once you have committed to a 200.

Why streaming exists#

The measurement that matters is time to first token, not total time. A response of five hundred tokens at fifty tokens a second takes ten seconds either way; unstreamed, the user waits ten seconds and then reads, and streamed, they start reading after perhaps four hundred milliseconds and the rest arrives faster than they can read it. Perceived latency collapses even though real latency is unchanged.

That is also the argument for streaming even when you do not display the tokens. A stream gives you a liveness signal: an unstreamed request that is hung and one that is working look identical until the timeout, whereas a stream that has stopped producing deltas is visibly stuck, and you can fail over while the user is still watching something happen.

Consuming a stream#

Providers use server-sent events, which is a text format over ordinary HTTP: lines of field-and-value, with a blank line ending each event. It is simple enough to parse by hand, and worth doing once because the thing that breaks is not the format but the chunking.

def parse_sse(chunks):
    """Lines of field: value, a blank line ends an event. The buffer is the
    whole trick -- an event can be split across any number of chunks."""
    buffer, events = "", []
    for chunk in chunks:
        buffer += chunk
        while "\n\n" in buffer:
            raw, buffer = buffer.split("\n\n", 1)
            event = {}
            for line in raw.splitlines():
                if not line or line.startswith(":"):
                    continue                       # blank or comment
                field, _, value = line.partition(":")
                event[field.strip()] = value.strip()
            if event:
                events.append(event)
    return events, buffer

# A provider does not send one event per chunk. This is the realistic case:
chunks = [
    'event: delta\ndata: {"text": "Hel',
    'lo"}\n\nevent: delta\nda',
    'ta: {"text": " world"}\n\nevent: done\ndata: [DONE]\n\n',
]
events, leftover = parse_sse(chunks)
for e in events:
    print(e)

naive = [c for c in chunks if c.startswith("event:")]
print("naive parser found", len(naive), "events instead of", len(events))

# {'event': 'delta', 'data': '{"text": "Hello"}'}
# {'event': 'delta', 'data': '{"text": " world"}'}
# {'event': 'done', 'data': '[DONE]'}
# naive parser found 1 events instead of 3

In practice the provider SDK does this for you and you iterate over typed events. Write it once anyway: the day a stream misbehaves, knowing that the transport is line-delimited text with a buffering rule is the difference between reading the raw bytes and guessing.

What it breaks#

Streaming removes the moment where you had the whole response and could decide what to do with it. Several things you were relying on lived in that moment.

what you lose            why                        what to do instead
validating the output    it arrives in pieces       validate at the end, or buffer
a moderation pass        nothing to scan yet        stream to a buffer, then release
retrying cleanly         the user saw half of it    only retry before the first token
exact token counts       usage arrives last         read the final usage event
structured output        partial JSON is invalid    parse incrementally, or buffer
caching the response     there is no response yet   assemble, then cache

Errors that arrive mid-stream#

This is the part that catches people. An unstreamed call either succeeds or fails, and you find out before responding. A streamed call sends 200 and a set of headers immediately, then fails after eight hundred tokens — a rate limit reached mid-generation, an overloaded backend, a dropped connection. The status code has already been sent and cannot be taken back.

• Errors arrive as an event in the stream, not as an HTTP status. Handle the error event type explicitly; a parser that only knows about deltas will treat it as an unfinished response.
• A stream that simply stops is indistinguishable from one that finished, unless you require the terminal event. Treat a stream that ends without it as a failure.
• Retrying after partial output means the user sees the beginning twice. Either retry only before the first token has been sent, or replace the whole message in the interface rather than appending.
• Your own server must forward the failure to the client somehow. The usual answer is an error event in your own stream plus a visible state in the interface, because you cannot change a 200 into a 500.
• Set an inactivity timeout, not just a total one. A stream producing nothing for thirty seconds is dead, and a total timeout of ten minutes will not notice for nine of them.

The inactivity timeout is the one most often missing, and it is also the one that turns a provider incident into a pile of stuck connections holding workers open. Time the gap between deltas rather than the whole request, and treat an exceeded gap as a failed stream you can fall back from — which is possible precisely because you are still connected and still producing something the user can see.

Streaming does not make a response faster. It makes the wait tolerable, which to a user amounts to much the same thing. This section builds it properly end to end, because the naive version breaks in specific and predictable places. The browser must not hold your API key, so the stream passes through your server. Some proxies buffer the whole response and deliver it in one piece, so it works locally and arrives all at once in production. Cancelling has to actually stop the upstream call or you pay for tokens nobody reads, and tool calls arrive in fragments that need reassembling.

The shape of it#

model API  --stream-->  your server  --stream-->  browser
                          (auth, logging,         (renders as
                           limits, keys)           text arrives)

The browser should never hold your API key, so the stream passes through your server. Major model APIs stream using server-sent events: a long-lived HTTP response that sends small text events as they are ready. It is plain HTTP, so it works through most infrastructure, with one catch below.

Forwarding a stream#

import json
from fastapi import FastAPI
from fastapi.responses import StreamingResponse
from pydantic import BaseModel

app = FastAPI()

class ChatRequest(BaseModel):
    messages: list

async def model_stream(messages):        # your SDK call, stubbed so this runs
    for text in ["Hel", "lo", " world"]:
        yield text

def sse(payload):
    """One event: a data line, then a blank line. The blank line is the frame."""
    return f"data: {json.dumps(payload)}\n\n"

@app.post("/chat")
async def chat(req: ChatRequest):
    async def events():
        async for text in model_stream(req.messages):
            yield sse({"text": text})
        yield "data: [DONE]\n\n"
    return StreamingResponse(events(), media_type="text/event-stream")

print(repr(sse({"text": "Hel"})))

# 'data: {"text": "Hel"}\n\n'

Two details in that handler are the whole job. The events are generated lazily, so nothing is assembled before it is sent — a handler that builds a list and returns it has a streaming content type and no streaming. And the media type is text/event-stream, which is what tells every layer between you and the browser that this response is not a document. Parsing the events on the way in is covered in the first section of this topic, which writes an SSE parser and shows what happens when an event is split across two network reads; this section assumes that parser and deals with the pipe it runs through.

The proxy that ruins it#

Some proxies and load balancers buffer responses and send them in one piece when complete. Your stream works locally and arrives all at once in production. Check each hop. For nginx, response buffering has to be turned off for this route, and a header such as X-Accel-Buffering: no does that per response.

Finding which hop is doing it is a measurement rather than an argument. Call the application directly, then through each layer in turn, with a client that timestamps every read rather than waiting for the whole body. Two numbers separate the cases: how many reads the client made, and how long after the request the first one arrived. A stream produces dozens of reads spread across the response; a buffered stream produces one read at the very end, holding exactly the same bytes.

def record(prefill, rate, n, buffered=False):
    """(seconds since the request was sent, deltas in that read). Recorded by a
    client that timestamps every read; buffered=True is what a proxy gives you."""
    end = prefill + n / rate
    if buffered:
        return [(round(end, 3), n)]
    return [(round(prefill + i / rate, 3), 1) for i in range(n)]

def stats(trace):
    ttft, last = trace[0][0], trace[-1][0]
    deltas = sum(n for _, n in trace)
    span = last - ttft
    return ttft, last, deltas, (deltas / span if span > 0 else None), len(trace)

print(f"{'hop':<12}{'ttft':>8}{'total':>8}{'deltas':>8}{'reads':>7}{'tok/s':>8}  verdict")
for name, trace in [("direct", record(0.42, 25, 40)),
                    ("via proxy", record(0.42, 25, 40, buffered=True))]:
    ttft, last, deltas, tps, reads = stats(trace)
    verdict = "streaming" if reads > 1 else "buffered, one read at the end"
    rate = f"{tps:.1f}" if tps else "-"
    print(f"{name:<12}{ttft:>7.2f}s{last:>7.2f}s{deltas:>8}{reads:>7}{rate:>8}  {verdict}")

# hop             ttft   total  deltas  reads   tok/s  verdict
# direct         0.42s   1.98s      40     40    25.6  streaming
# via proxy      2.02s   2.02s      40      1       -  buffered, one read at the end

The usual culprits, in the order they are worth checking: nginx or another reverse proxy with response buffering on by default; compression, because gzip wants a block of bytes to compress and will hold them until it has one; a CDN in front of the origin treating the response as a cacheable document; and a serverless platform whose function response model returns a whole body rather than a stream. Each is fixed in a different place, which is why the measurement above is worth more than a list of settings.

What streaming actually buys#

print(f"{'reply tokens':>13}{'unstreamed':>12}{'streamed':>10}{'saved':>8}")
for n in (40, 200, 800):
    total = 0.42 + n / 25
    print(f"{n:>13}{total:>11.1f}s{0.42:>9.2f}s{total - 0.42:>7.1f}s")

#  reply tokens  unstreamed  streamed   saved
#            40        2.0s     0.42s    1.6s
#           200        8.4s     0.42s    8.0s
#           800       32.4s     0.42s   32.0s

Time to first token is dominated by prefill, which scales with how much prompt the model has to read, so a long retrieved context pushes the one number streaming was supposed to protect. Watch it as a percentile rather than a mean — the median user sees the healthy number and the tail is where the abandoned sessions are — and watch total time separately, because it is unchanged by streaming and it is the only figure that matters to anything automated downstream.

Cancelling#

Users press stop, close the tab, or send a new message. When the browser disconnects, your server should stop reading from the model and close that connection too, otherwise the model keeps generating text nobody will see. You are generally billed for tokens generated up to the point the request is cancelled, so stopping promptly saves money.

The failure is quiet, because a server that ignores the disconnect and drains the upstream stream into nothing behaves correctly in every way a test would check. Here is what it costs, with a model that produces four hundred tokens and a reader who leaves after thirty.

import asyncio

async def upstream(queue, n_tokens, meter):
    """Stands in for the model. Every token it produces is billed."""
    for i in range(n_tokens):
        meter["generated"] += 1
        await queue.put(f"t{i} ")
    await queue.put(None)

async def forward(read_n, propagate):
    meter = {"generated": 0}
    queue = asyncio.Queue(maxsize=1)          # the model is paced by the reader
    task = asyncio.create_task(upstream(queue, 400, meter))
    for _ in range(read_n):                   # the browser then disconnects
        await queue.get()
    if propagate:
        task.cancel()
        try:
            await task
        except asyncio.CancelledError:
            pass
    else:
        while await queue.get() is not None:  # your server drains it anyway
            pass
        await task
    return meter["generated"]

OUT_PER_M = 15.00                             # usd per million output tokens
for propagate in (True, False):
    billed = asyncio.run(forward(30, propagate))
    label = "cancel propagated" if propagate else "client gone, upstream not told"
    wasted = billed - 30
    print(f"{label:30}{billed:>6} billed{wasted:>6} wasted"
          f"{wasted * OUT_PER_M:>10.0f} usd per million such requests")

# cancel propagated                 31 billed     1 wasted        15 usd per million such requests
# client gone, upstream not told   400 billed   370 wasted      5550 usd per million such requests

In a real handler the disconnect arrives as an event from the framework — Starlette calls it await request.is_disconnected, and in Node it is the request close event — and the response is to cancel the task holding the upstream connection, not merely to stop writing to the client. The SDK streaming context manager does the right thing on exit, so the pattern that works is to keep the upstream call inside a task you own and can cancel, rather than inside a generator someone else is iterating.

Tool calls arrive in pieces#

When the model calls a tool during a stream, the arguments arrive as fragments of JSON. You cannot parse them until the fragment stream for that call has finished. Accumulate, then parse, then run the tool.

import json

DELTAS = [
    {"index": 0, "name": "search_orders", "args": ""},
    {"index": 0, "args": '{"customer'},
    {"index": 0, "args": '_id": "c-8'},
    {"index": 0, "args": '81", "status"'},
    {"index": 0, "args": ': "refunded"}'},
    {"index": 0, "stop": True},
]

def accumulate(deltas):
    calls, naive_failures = {}, 0
    for d in deltas:
        call = calls.setdefault(d["index"], {"name": None, "args": ""})
        if d.get("name"):
            call["name"] = d["name"]
        call["args"] += d.get("args", "")
        if d.get("args"):
            try:
                json.loads(call["args"])          # what parsing early looks like
            except json.JSONDecodeError:
                naive_failures += 1
        if d.get("stop"):
            call["parsed"] = json.loads(call["args"])
    return calls, naive_failures

calls, failures = accumulate(DELTAS)
print(calls[0]["name"], calls[0]["parsed"])
print(f"{failures} of {sum(1 for d in DELTAS if d.get('args'))} fragments were not valid JSON on their own")

# search_orders {'customer_id': 'c-881', 'status': 'refunded'}
# 3 of 4 fragments were not valid JSON on their own

The interface question that follows is what to show while that is happening. Showing the fragments is meaningless, and showing nothing loses the liveness streaming bought you, so the usual answer is to render the tool name as soon as it arrives — 'searching orders' — and the result when the tool returns. That also gives you somewhere honest to put a tool call that fails.

Measure what users feel#

• Time to first token: how long until anything appears. This is what streaming improves.
• Tokens per second after that: how fast text flows.
• Total time: unchanged by streaming, and what matters for anything automated.
• Cancellation rate, and how far into the reply people leave. It sizes the waste above and tells you whether your answers are too long.
• Streams that ended without a terminal event, as a rate. A stream that stops looks exactly like one that finished unless you require the last event and count the ones that never sent it.