Failures, parallel calls and keeping results small

A tool that behaves in a demo behaves differently under real traffic: arguments arrive malformed, an API times out, a query comes back with half a megabyte of rows. This section covers what to do in each case, and the answer is usually not what an exception handler would normally do. Done well, these are recoverable situations the model works around on its own; done badly, one of them ends the run or quietly fills the window with output nobody will read.

An error is a result, not a crash#

When a tool fails, the natural instinct is to let the exception propagate. That ends the run. Usually the better move is to hand the failure back to the model as the tool's result, because the model can often recover: fix a typo in an argument, try a different approach, or tell the user what went wrong.

def run_tool(call):
    try:
        return {"ok": True, "data": TOOL_FUNCTIONS[call.name](**call.arguments)}
    except KeyError:
        return {"ok": False, "error": f"No tool named {call.name}"}
    except Exception as e:
        return {"ok": False, "error": f"{type(e).__name__}: {e}"}

Some failures should still stop everything. An authentication error will not be fixed by the model retrying, and letting it try ten times wastes money and time. Distinguish recoverable from terminal deliberately.

Several calls at once#

A model can request more than one tool in a single turn when the calls are independent. Run them concurrently, since they are usually network-bound, and then return every result together in one message.

results = await asyncio.gather(*(run_tool(c) for c in response.tool_calls))

messages.append(response.message)
for call, result in zip(response.tool_calls, results):
    messages.append({
        "role": "tool",
        "tool_call_id": call.id,
        "content": json.dumps(result),
    })

Validate before you execute#

The model generates arguments, so treat them as untrusted input. Schema constraints catch type errors, but they do not catch a valid-looking value that is wrong, and they certainly do not catch a destructive one.

def delete_records(query: str, confirm: bool = False):
    if not confirm:
        return {"ok": False,
                "error": "Destructive action needs confirm=true and human approval"}
    ...

Keep results small#

Tool results go into the context and stay there for the rest of the conversation, being resent every turn. A tool that returns a 5,000-token payload has just made every subsequent turn more expensive, permanently.

• Return the fields the model needs, not the whole database row.
• Cap list results, and say so in the response, for example that ten of forty-two matches are shown.
• Summarise long text before returning it, or return an identifier the model can use to fetch more.
• Prefer several small focused tools over one that returns everything.