Parameters that change the output

Alongside your messages you can send a few settings that change how the model writes: how adventurous it is allowed to be, how long the reply may run, and what standing instructions it should follow throughout. This section is about what each one really does, so that you set them deliberately rather than copying them out of an example. The two most often misused are temperature, which does not make a model more accurate, and the length ceiling, which cuts a reply off rather than asking for a shorter one.

Temperature#

At each step the model has a probability for every possible next token. Temperature changes how sharply it favours the likely ones. Low temperature makes it pick the most probable token almost every time. High temperature flattens the distribution so less likely tokens get a chance.

• 0 to 0.3: extraction, classification, anything you will parse, anything you will evaluate.
• 0.7 to 1.0: writing, brainstorming, anywhere variety is the point.
• Above about 1.5: usually degrades into nonsense.

"Name one colour."     t=0.0  ->  Blue, Blue, Blue
"Name one colour."     t=2.0  ->  Blue, Blue, Blue
"Name any animal."     t=0.0  ->  Elephant, Lion, Lion, Elephant

Temperature 0 is not deterministic#

People assume temperature 0 means the same input always gives the same output. On a hosted API it does not. Requests are batched together on GPUs, floating point addition is not perfectly associative, and which other requests share your batch changes the result slightly. Different hardware and expert routing add more variation.

This matters most when you write evals. An exact-match assertion on generated text will flake, get marked as unreliable, and eventually be deleted. Assert on structure and properties instead, or run repeatedly and check a pass rate.

max_tokens#

This caps the length of the reply. It does not tell the model to be brief. The model writes as though it had unlimited room and generation is cut off when the cap is reached, usually mid-sentence.

# Wrong way to get a short answer
client.chat(messages, max_tokens=50)

# Right way
messages = [
    {"role": "system", "content": "Answer in at most two sentences."},
    ...
]

Still set max_tokens, as a safety net against a runaway response. Set it comfortably above what you expect, and check stop_reason to find out whether you hit it.

System prompts#

The system message sets standing instructions for the whole conversation: the role, the format, the tone, what to do when unsure. Keep it stable, because a stable prefix is what makes prompt caching work later.

SYSTEM = """You extract invoice data.

Return only JSON matching the schema you are given.
If a field is not present in the text, use null rather than guessing.
Never invent a number."""