Streaming responses interview questions

7 questions across 2 sections, in the order you would learn them. Every answer here is one that works — but an answer you have read is not an answer you can give, so use Test yourself and say it out loud first.

Streaming a response

Read the section that teaches this

If this comes up in an interview

Does streaming reduce latency?

It reduces perceived latency by showing tokens as they are generated. Total time to the complete answer and total cost are unchanged.

What does streaming complicate?

Validation, since you cannot check a schema until the stream ends. Error handling, since a failure can arrive after the user has seen half an answer. Retries and output moderation, both of which assume you can act before anything is shown.

Would you stream a response that must be valid JSON?

There is little point. You cannot use it until it is complete, so streaming only gives you a progress indicator. Stream prose to users, and buffer anything you intend to parse.

Then they push further

What if the user needs to see progress?

Stream status updates or a separate readable field while the structured result is produced, and parse the JSON only once it is complete. Stream partial JSON with a tolerant parser only if the interface truly needs it.

What gets marked up. Marked up for noting that JSON cannot be validated until it is complete, so streaming helps the person watching but not the program consuming the result.

Does streaming reduce latency?What does streaming complicate?Would you stream a response that must be valid JSON?

Streaming from the model to the screen

Read the section that teaches this

If this comes up in an interview

Your streaming chat works locally but the whole answer appears at once in production. Why?

Something between the server and the browser is buffering the response, usually a reverse proxy or load balancer. Turn off buffering for that route, check compression settings, and test each hop.

Then they push further

How would you find which hop is buffering?

Call each layer directly in turn, the application, then through the proxy, then through the load balancer, with a client that prints events with timestamps. The first layer where events arrive together is the culprit.

What gets marked up. Marked up for going straight to buffering in intermediaries and having a method to find it.

What happens when a user cancels a streamed answer?

The server should notice the client disconnect and cancel the upstream request, so generation stops. Otherwise the model keeps producing tokens you pay for and nobody reads.

What metric does streaming improve?

Time to first token, which is what users perceive as speed. Total generation time is unchanged.

How do you handle a tool call that arrives in a stream?

The arguments come as fragments of a JSON string, split at points that respect nothing about JSON, so you accumulate them keyed by the call index and parse once the block is finished. Parsing each fragment as it arrives fails on nearly all of them. Show the tool name in the interface as soon as it arrives, and the result when the tool returns, so the user sees progress without seeing half-formed JSON.

What gets marked up. Marked up for keying by index, which is what makes parallel tool calls work, and for knowing that the name arrives before the arguments.

Your streaming chat works locally but the whole answer appears at once in production. Why?What happens when a user cancels a streamed answer?What metric does streaming improve?How do you handle a tool call that arrives in a stream?

These are the questions for one topic. The whole question bank covers every topic on the site.