Asked about Parameters that change the output

Is temperature 0 deterministic?

An answer that works

Not on a hosted API. Batched GPU execution, non-associative floating point addition, expert routing and varying hardware all introduce small differences. This is why exact-match eval assertions on generated text flake.

Then they push further

If you need the same output for the same input, what do you do?

Cache results keyed by the exact input and settings, and design so small variations do not matter, for example by validating the output rather than comparing it to a stored string.

What gets marked up

Marked up for knowing hosted inference varies even at temperature 0, because of batching and floating point effects, and for designing around that rather than relying on it.

Reading an answer is not being able to give one. The section that teaches this is Parameters that change the output, 2 hours of The Messages API.

Learn this properly

Also asked about The Messages API

What are the roles in a chat request and what is each for?What is stop_reason and why check it?Does max_tokens make the model give shorter answers?Are LLM APIs stateful or stateless?Why does a long conversation get expensive faster than you would expect?How would you keep a long-running chat affordable?What does temperature do?How do you make a model give shorter answers?

All 10 questions on The Messages API