Tool calling

Everything so far has ended with the model producing an answer. Tool calling is how it gets to do more than that: you describe the actions available, the model asks for one, and your own program carries it out and reports back. This section follows that round trip a step at a time, because the division of labour within it is what people misread, and misreading it leads to designs that assume the model can reach out into the world by itself.

The model asks, your code acts#

This is the point people misunderstand. The model cannot run code, call an API or touch a database. It can only produce text. Tool calling is a convention where part of that text is a structured request, which your program reads and acts on.

1. You send:     messages + a list of available tools
2. Model replies: "call get_weather with {city: 'Berlin'}"
3. YOUR code runs get_weather("Berlin")  ->  "18C, cloudy"
4. You send:     everything again, plus that result
5. Model replies: "It is 18 degrees and cloudy in Berlin."

Note steps 4 and 5. A tool call always needs a second request, because the model has to see the result before it can use it. One user question becomes at least two API calls.

Describing a tool#

A tool definition is a name, a description and a schema for its arguments. The description is not documentation for you. It is the only thing the model uses to decide whether this tool is the right one, so it should say when to use it, not just what it does.

{
    "name": "search_orders",
    "description": (
        "Find a customer's orders by email address. "
        "Use this when the user asks about order status, delivery "
        "or refunds. Returns at most 10 of the most recent orders."
    ),
    "parameters": {
        "type": "object",
        "properties": {
            "email": {
                "type": "string",
                "description": "The customer's email address",
            },
            "status": {
                "type": "string",
                "enum": ["pending", "shipped", "delivered"],
                "description": "Optional filter by order status",
            },
        },
        "required": ["email"],
    },
}

The round trip in code#

messages = [{"role": "user", "content": "Where is my order?"}]

response = client.chat(messages, tools=TOOLS)

if response.stop_reason == "tool_use":
    for call in response.tool_calls:
        result = TOOL_FUNCTIONS[call.name](**call.arguments)

        messages.append(response.message)          # what the model asked
        messages.append({                          # what it got back
            "role": "tool",
            "tool_call_id": call.id,
            "content": json.dumps(result),
        })

    response = client.chat(messages, tools=TOOLS)  # ask again, with results

print(response.text)

Both appends are required. The model's request has to stay in the history, or the result arrives with nothing to attach to and the model loses track of what it asked.

A tool that behaves in a demo behaves differently under real traffic: arguments arrive malformed, an API times out, a query comes back with half a megabyte of rows. This section covers what to do in each case, and the answer is usually not what an exception handler would normally do. Done well, these are recoverable situations the model works around on its own; done badly, one of them ends the run or quietly fills the window with output nobody will read.

An error is a result, not a crash#

When a tool fails, the natural instinct is to let the exception propagate. That ends the run. Usually the better move is to hand the failure back to the model as the tool's result, because the model can often recover: fix a typo in an argument, try a different approach, or tell the user what went wrong.

def run_tool(call):
    try:
        return {"ok": True, "data": TOOL_FUNCTIONS[call.name](**call.arguments)}
    except KeyError:
        return {"ok": False, "error": f"No tool named {call.name}"}
    except Exception as e:
        return {"ok": False, "error": f"{type(e).__name__}: {e}"}

Some failures should still stop everything. An authentication error will not be fixed by the model retrying, and letting it try ten times wastes money and time. Distinguish recoverable from terminal deliberately.

Several calls at once#

A model can request more than one tool in a single turn when the calls are independent. Run them concurrently, since they are usually network-bound, and then return every result together in one message.

results = await asyncio.gather(*(run_tool(c) for c in response.tool_calls))

messages.append(response.message)
for call, result in zip(response.tool_calls, results):
    messages.append({
        "role": "tool",
        "tool_call_id": call.id,
        "content": json.dumps(result),
    })

Validate before you execute#

The model generates arguments, so treat them as untrusted input. Schema constraints catch type errors, but they do not catch a valid-looking value that is wrong, and they certainly do not catch a destructive one.

def delete_records(query: str, confirm: bool = False):
    if not confirm:
        return {"ok": False,
                "error": "Destructive action needs confirm=true and human approval"}
    ...

Keep results small#

Tool results go into the context and stay there for the rest of the conversation, being resent every turn. A tool that returns a 5,000-token payload has just made every subsequent turn more expensive, permanently.

• Return the fields the model needs, not the whole database row.
• Cap list results, and say so in the response, for example that ten of forty-two matches are shown.
• Summarise long text before returning it, or return an identifier the model can use to fetch more.
• Prefer several small focused tools over one that returns everything.