Streaming from the model to the screen

Streaming does not make a response faster. It makes the wait tolerable, which to a user amounts to much the same thing. This section builds it properly end to end, because the naive version breaks in specific and predictable places. The browser must not hold your API key, so the stream passes through your server. Some proxies buffer the whole response and deliver it in one piece, so it works locally and arrives all at once in production. Cancelling has to actually stop the upstream call or you pay for tokens nobody reads, and tool calls arrive in fragments that need reassembling.

The shape of it#

model API  --stream-->  your server  --stream-->  browser
                          (auth, logging,         (renders as
                           limits, keys)           text arrives)

The browser should never hold your API key, so the stream passes through your server. Major model APIs stream using server-sent events: a long-lived HTTP response that sends small text events as they are ready. It is plain HTTP, so it works through most infrastructure, with one catch below.

Forwarding a stream#

import json
from fastapi import FastAPI
from fastapi.responses import StreamingResponse
from pydantic import BaseModel

app = FastAPI()

class ChatRequest(BaseModel):
    messages: list

async def model_stream(messages):        # your SDK call, stubbed so this runs
    for text in ["Hel", "lo", " world"]:
        yield text

def sse(payload):
    """One event: a data line, then a blank line. The blank line is the frame."""
    return f"data: {json.dumps(payload)}\n\n"

@app.post("/chat")
async def chat(req: ChatRequest):
    async def events():
        async for text in model_stream(req.messages):
            yield sse({"text": text})
        yield "data: [DONE]\n\n"
    return StreamingResponse(events(), media_type="text/event-stream")

print(repr(sse({"text": "Hel"})))

# 'data: {"text": "Hel"}\n\n'

Two details in that handler are the whole job. The events are generated lazily, so nothing is assembled before it is sent — a handler that builds a list and returns it has a streaming content type and no streaming. And the media type is text/event-stream, which is what tells every layer between you and the browser that this response is not a document. Parsing the events on the way in is covered in the first section of this topic, which writes an SSE parser and shows what happens when an event is split across two network reads; this section assumes that parser and deals with the pipe it runs through.

The proxy that ruins it#

Some proxies and load balancers buffer responses and send them in one piece when complete. Your stream works locally and arrives all at once in production. Check each hop. For nginx, response buffering has to be turned off for this route, and a header such as X-Accel-Buffering: no does that per response.

Finding which hop is doing it is a measurement rather than an argument. Call the application directly, then through each layer in turn, with a client that timestamps every read rather than waiting for the whole body. Two numbers separate the cases: how many reads the client made, and how long after the request the first one arrived. A stream produces dozens of reads spread across the response; a buffered stream produces one read at the very end, holding exactly the same bytes.

def record(prefill, rate, n, buffered=False):
    """(seconds since the request was sent, deltas in that read). Recorded by a
    client that timestamps every read; buffered=True is what a proxy gives you."""
    end = prefill + n / rate
    if buffered:
        return [(round(end, 3), n)]
    return [(round(prefill + i / rate, 3), 1) for i in range(n)]

def stats(trace):
    ttft, last = trace[0][0], trace[-1][0]
    deltas = sum(n for _, n in trace)
    span = last - ttft
    return ttft, last, deltas, (deltas / span if span > 0 else None), len(trace)

print(f"{'hop':<12}{'ttft':>8}{'total':>8}{'deltas':>8}{'reads':>7}{'tok/s':>8}  verdict")
for name, trace in [("direct", record(0.42, 25, 40)),
                    ("via proxy", record(0.42, 25, 40, buffered=True))]:
    ttft, last, deltas, tps, reads = stats(trace)
    verdict = "streaming" if reads > 1 else "buffered, one read at the end"
    rate = f"{tps:.1f}" if tps else "-"
    print(f"{name:<12}{ttft:>7.2f}s{last:>7.2f}s{deltas:>8}{reads:>7}{rate:>8}  {verdict}")

# hop             ttft   total  deltas  reads   tok/s  verdict
# direct         0.42s   1.98s      40     40    25.6  streaming
# via proxy      2.02s   2.02s      40      1       -  buffered, one read at the end

The usual culprits, in the order they are worth checking: nginx or another reverse proxy with response buffering on by default; compression, because gzip wants a block of bytes to compress and will hold them until it has one; a CDN in front of the origin treating the response as a cacheable document; and a serverless platform whose function response model returns a whole body rather than a stream. Each is fixed in a different place, which is why the measurement above is worth more than a list of settings.

What streaming actually buys#

print(f"{'reply tokens':>13}{'unstreamed':>12}{'streamed':>10}{'saved':>8}")
for n in (40, 200, 800):
    total = 0.42 + n / 25
    print(f"{n:>13}{total:>11.1f}s{0.42:>9.2f}s{total - 0.42:>7.1f}s")

#  reply tokens  unstreamed  streamed   saved
#            40        2.0s     0.42s    1.6s
#           200        8.4s     0.42s    8.0s
#           800       32.4s     0.42s   32.0s

Time to first token is dominated by prefill, which scales with how much prompt the model has to read, so a long retrieved context pushes the one number streaming was supposed to protect. Watch it as a percentile rather than a mean — the median user sees the healthy number and the tail is where the abandoned sessions are — and watch total time separately, because it is unchanged by streaming and it is the only figure that matters to anything automated downstream.

Cancelling#

Users press stop, close the tab, or send a new message. When the browser disconnects, your server should stop reading from the model and close that connection too, otherwise the model keeps generating text nobody will see. You are generally billed for tokens generated up to the point the request is cancelled, so stopping promptly saves money.

The failure is quiet, because a server that ignores the disconnect and drains the upstream stream into nothing behaves correctly in every way a test would check. Here is what it costs, with a model that produces four hundred tokens and a reader who leaves after thirty.

import asyncio

async def upstream(queue, n_tokens, meter):
    """Stands in for the model. Every token it produces is billed."""
    for i in range(n_tokens):
        meter["generated"] += 1
        await queue.put(f"t{i} ")
    await queue.put(None)

async def forward(read_n, propagate):
    meter = {"generated": 0}
    queue = asyncio.Queue(maxsize=1)          # the model is paced by the reader
    task = asyncio.create_task(upstream(queue, 400, meter))
    for _ in range(read_n):                   # the browser then disconnects
        await queue.get()
    if propagate:
        task.cancel()
        try:
            await task
        except asyncio.CancelledError:
            pass
    else:
        while await queue.get() is not None:  # your server drains it anyway
            pass
        await task
    return meter["generated"]

OUT_PER_M = 15.00                             # usd per million output tokens
for propagate in (True, False):
    billed = asyncio.run(forward(30, propagate))
    label = "cancel propagated" if propagate else "client gone, upstream not told"
    wasted = billed - 30
    print(f"{label:30}{billed:>6} billed{wasted:>6} wasted"
          f"{wasted * OUT_PER_M:>10.0f} usd per million such requests")

# cancel propagated                 31 billed     1 wasted        15 usd per million such requests
# client gone, upstream not told   400 billed   370 wasted      5550 usd per million such requests

In a real handler the disconnect arrives as an event from the framework — Starlette calls it await request.is_disconnected, and in Node it is the request close event — and the response is to cancel the task holding the upstream connection, not merely to stop writing to the client. The SDK streaming context manager does the right thing on exit, so the pattern that works is to keep the upstream call inside a task you own and can cancel, rather than inside a generator someone else is iterating.

Tool calls arrive in pieces#

When the model calls a tool during a stream, the arguments arrive as fragments of JSON. You cannot parse them until the fragment stream for that call has finished. Accumulate, then parse, then run the tool.

import json

DELTAS = [
    {"index": 0, "name": "search_orders", "args": ""},
    {"index": 0, "args": '{"customer'},
    {"index": 0, "args": '_id": "c-8'},
    {"index": 0, "args": '81", "status"'},
    {"index": 0, "args": ': "refunded"}'},
    {"index": 0, "stop": True},
]

def accumulate(deltas):
    calls, naive_failures = {}, 0
    for d in deltas:
        call = calls.setdefault(d["index"], {"name": None, "args": ""})
        if d.get("name"):
            call["name"] = d["name"]
        call["args"] += d.get("args", "")
        if d.get("args"):
            try:
                json.loads(call["args"])          # what parsing early looks like
            except json.JSONDecodeError:
                naive_failures += 1
        if d.get("stop"):
            call["parsed"] = json.loads(call["args"])
    return calls, naive_failures

calls, failures = accumulate(DELTAS)
print(calls[0]["name"], calls[0]["parsed"])
print(f"{failures} of {sum(1 for d in DELTAS if d.get('args'))} fragments were not valid JSON on their own")

# search_orders {'customer_id': 'c-881', 'status': 'refunded'}
# 3 of 4 fragments were not valid JSON on their own

The interface question that follows is what to show while that is happening. Showing the fragments is meaningless, and showing nothing loses the liveness streaming bought you, so the usual answer is to render the tool name as soon as it arrives — 'searching orders' — and the result when the tool returns. That also gives you somewhere honest to put a tool call that fails.

Measure what users feel#

• Time to first token: how long until anything appears. This is what streaming improves.
• Tokens per second after that: how fast text flows.
• Total time: unchanged by streaming, and what matters for anything automated.
• Cancellation rate, and how far into the reply people leave. It sizes the waste above and tells you whether your answers are too long.
• Streams that ended without a terminal event, as a rate. A stream that stops looks exactly like one that finished unless you require the last event and count the ones that never sent it.