The Messages API interview questions
10 questions across 3 sections, in the order you would learn them. Every answer here is one that works — but an answer you have read is not an answer you can give, so use Test yourself and say it out loud first.
Making a call
Read the section that teaches thisIf this comes up in an interview
What are the roles in a chat request and what is each for?
system carries your instructions and is not shown to the user. user is what the person said. assistant, called model by some providers, is what the model said on previous turns. The list is the whole conversation.
What is stop_reason and why check it?
It says why generation ended: finished naturally, hit the token limit, or requested a tool. Hitting the limit truncates the output without raising an exception, so without checking it you get silently incomplete data.
Does max_tokens make the model give shorter answers?
No. It is a hard cut-off, not an instruction. The model writes as though unconstrained and gets chopped mid-word. To get a shorter answer, ask for one in the prompt.
The model has no memory
Read the section that teaches thisIf this comes up in an interview
Are LLM APIs stateful or stateless?
Stateless. The model retains nothing between calls. Conversation is created by the caller resending the full history each turn. Some products offer a session id, but underneath they are storing and resending the history for you.
Why does a long conversation get expensive faster than you would expect?
Every turn resends all previous turns, so total tokens grow with the square of the number of turns rather than linearly. Most of that resent text is the model's own previous output, which is typically longer than the user's messages.
Then they push further
Roughly how much faster?
If each turn adds a similar amount, total input over n turns grows with the square of n, because every turn resends everything before it. Twice as many turns costs roughly four times as much input.
What would you do about it?
Cache the stable prefix, trim or summarise older turns, and drop bulky tool results once they are no longer needed.
What gets marked up. Strong answers explain that the API is stateless so the whole history is sent every turn, and can reason about the growth.
How would you keep a long-running chat affordable?
Prompt caching on the stable prefix first, since it is free quality-wise. Then summarise or drop older turns, keeping a short running summary. Then retrieve only relevant history instead of carrying all of it. Increasing the context window is the last resort, not the first.
Parameters that change the output
Read the section that teaches thisIf this comes up in an interview
What does temperature do?
It scales the probability distribution before a token is sampled. Low values concentrate probability on the most likely token, high values spread it out. It can only produce variety where the model was already uncertain.
Is temperature 0 deterministic?
Not on a hosted API. Batched GPU execution, non-associative floating point addition, expert routing and varying hardware all introduce small differences. This is why exact-match eval assertions on generated text flake.
Then they push further
If you need the same output for the same input, what do you do?
Cache results keyed by the exact input and settings, and design so small variations do not matter, for example by validating the output rather than comparing it to a stored string.
What gets marked up. Marked up for knowing hosted inference varies even at temperature 0, because of batching and floating point effects, and for designing around that rather than relying on it.
How do you make a model give shorter answers?
Ask for it in the prompt. max_tokens is a hard cut-off that truncates mid-sentence rather than an instruction to be concise. Keep max_tokens as a safety limit and check stop_reason to detect when it fires.
What belongs in the system prompt rather than the user message?
Anything constant across the conversation: role, output format, tone, and the rule for what to do when information is missing. Keeping it stable also lets prompt caching apply to it.
These are the questions for one topic. The whole question bank covers every topic on the site.