The Messages API

Talking to a model means sending some text over the internet and reading what comes back, and this is where that exchange gets pinned down: how the thing you send is put together, why each piece of it is labelled with a speaker, and how to read the reply properly rather than just printing it. What comes back carries more than the answer, including a field saying whether the model finished or simply ran out of room, and ignoring that field is how truncated text reaches users. Everything else you build sits on top of this one exchange.

A request is a list of messages#

You send a list. Each item has a role saying who is speaking, and some content. The model replies with one message.

# pip install anthropic
import os
from anthropic import Anthropic

client = Anthropic(api_key=os.environ["ANTHROPIC_API_KEY"])

response = client.messages.create(
    model="claude-sonnet-5",
    max_tokens=1024,
    messages=[{"role": "user", "content": "What is a stateless API?"}],
)

print(response.content[0].text)

The three roles#

• system: your instructions. Who the model is, what it should do, what format to use. Not shown to the user.
• user: what the person said.
• assistant: what the model said previously. Providers differ here; some call this role model rather than assistant.

messages = [
    {"role": "system",    "content": "You answer in one sentence."},
    {"role": "user",      "content": "What is Python?"},
    {"role": "assistant", "content": "A high-level programming language."},
    {"role": "user",      "content": "Who created it?"},
]

Reading the response#

Beyond the text, two fields matter on every response and most people ignore both.

response.text                      # the answer
response.usage.input_tokens        # what you were charged for sending
response.usage.output_tokens       # what you were charged for receiving
response.stop_reason               # why it stopped

stop_reason tells you whether you got the whole answer. The common values are a natural finish, hitting the maximum token limit, or the model asking to call a tool.

if response.stop_reason == "max_tokens":
    # The answer is cut off mid-sentence. No exception was raised.
    raise ValueError("truncated: raise max_tokens or shorten the task")

Chat products feel as though they remember you. The thing underneath remembers nothing at all, and this section is about what follows from that. You will prove it to yourself, then build a conversation the way every chat product does, by resending the entire history on every turn. The consequence that catches people out lands on the bill, because the twentieth turn is paying for the first nineteen all over again.

Every call is the first call#

The model remembers nothing between requests. Each call is a separate HTTP request that carries everything the model will know. Nothing is stored on the other end.

This is easy to accept and hard to believe, because chat products feel like they remember. Prove it to yourself in two calls.

ask("My name is Jack.")
# "Nice to meet you, Jack!"

ask("What is my name?")
# "I don't have access to your name."

So what is a conversation?#

Something you build. To make turn five work, you resend turns one to four along with the new question. The illusion of memory is entirely on your side of the wire.

history = []

def ask(text):
    history.append({"role": "user", "content": text})

    reply = client.chat(history)                     # send ALL of it

    history.append({"role": "assistant", "content": reply.text})
    return reply.text

Why this gets expensive#

Each turn resends everything before it. Turn one sends one message, turn ten sends nineteen. Total cost across a conversation grows with the square of its length, not in proportion to it.

Turn 1:   7 input tokens
Turn 2:  33 input tokens
Turn 3:  48 input tokens

There is a second surprise in those numbers. Model replies are usually longer than user questions, so most of what you are paying to resend is the model's own earlier output.

Every technique you will meet later, prompt caching, summarising old turns, retrieval instead of history, is an answer to this one problem.

Alongside your messages you can send a few settings that change how the model writes: how adventurous it is allowed to be, how long the reply may run, and what standing instructions it should follow throughout. This section is about what each one really does, so that you set them deliberately rather than copying them out of an example. The two most often misused are temperature, which does not make a model more accurate, and the length ceiling, which cuts a reply off rather than asking for a shorter one.

Temperature#

At each step the model has a probability for every possible next token. Temperature changes how sharply it favours the likely ones. Low temperature makes it pick the most probable token almost every time. High temperature flattens the distribution so less likely tokens get a chance.

• 0 to 0.3: extraction, classification, anything you will parse, anything you will evaluate.
• 0.7 to 1.0: writing, brainstorming, anywhere variety is the point.
• Above about 1.5: usually degrades into nonsense.

"Name one colour."     t=0.0  ->  Blue, Blue, Blue
"Name one colour."     t=2.0  ->  Blue, Blue, Blue
"Name any animal."     t=0.0  ->  Elephant, Lion, Lion, Elephant

Temperature 0 is not deterministic#

People assume temperature 0 means the same input always gives the same output. On a hosted API it does not. Requests are batched together on GPUs, floating point addition is not perfectly associative, and which other requests share your batch changes the result slightly. Different hardware and expert routing add more variation.

This matters most when you write evals. An exact-match assertion on generated text will flake, get marked as unreliable, and eventually be deleted. Assert on structure and properties instead, or run repeatedly and check a pass rate.

max_tokens#

This caps the length of the reply. It does not tell the model to be brief. The model writes as though it had unlimited room and generation is cut off when the cap is reached, usually mid-sentence.

# Wrong way to get a short answer
client.chat(messages, max_tokens=50)

# Right way
messages = [
    {"role": "system", "content": "Answer in at most two sentences."},
    ...
]

Still set max_tokens, as a safety net against a runaway response. Set it comfortably above what you expect, and check stop_reason to find out whether you hit it.

System prompts#

The system message sets standing instructions for the whole conversation: the role, the format, the tone, what to do when unsure. Keep it stable, because a stable prefix is what makes prompt caching work later.

SYSTEM = """You extract invoice data.

Return only JSON matching the schema you are given.
If a field is not present in the text, use null rather than guessing.
Never invent a number."""