Worked design: an agent acting on customer accounts

The previous design only had to return text. This one changes real things: money moves, addresses change, and a mistake is not a bad answer but a refund that should not have happened. This section works through that design starting where it has to start, with a table of what the agent may do alone, what needs a person to approve it, and what it may never do, and then covers the tools, the audit trail, and how you evaluate something whose output is a change to the world.

The prompt#

Design a support agent that can look up orders, issue refunds and update account details. The difficulty is that it changes real things, so mistakes cost money and trust.

Clarify, then size it#

Ask how many contacts a day, what proportion the agent is expected to close without a human, what the average refund is, and what the current cost of a support contact is. That last question surprises people and it is the one that sets every threshold in the design, because an approval gate is only sensible if a human review costs less than the mistake it prevents.

Twenty million orders a month, 4 per cent contact rate:

  tickets    20,000,000 x 0.04  =  800,000 a month
                                =  ~26,700 a day
  rate       26,700 / 86,400    =  0.31 conversations a second
             peak at 3x         =  ~1 a second
  calls      6 turns each       =  160,000 model calls a day
                                =  1.9 a second, ~5.6 at peak

Tokens per conversation, 6 turns:
  system prompt and tool schemas  1,800, resent every turn
  history growing ~600 a turn
  total in   6x1,800 + 600x(0+1+2+3+4+5)  =  19,800
  total out  6 x 150                      =     900

Scope the actions#

Now the table, and it comes before the architecture rather than after it, because it determines what the architecture has to support. Sort every action by what it costs when the agent is wrong, not by how hard it is to build. Reading is free to get wrong. Resending a receipt to the account's own address is an annoyance. A refund is money. Changing payment details is fraud, and no eval score makes it acceptable for the agent to do that alone.

Read only, no approval      look up order, check status, read history
Low risk, no approval       resend a receipt, update a preference
Consequential, approval     refund above a threshold, change an address
Never                       delete an account, change payment details

The threshold in row three is a business number and you should say so. Set it where the expected loss from an automatic mistake equals the cost of a human glance. If a review costs a pound of someone's time and the agent is wrong one refund in two hundred, automatic refunds up to about two hundred pounds break even. Give the interviewer the method rather than a figure; the figure is theirs.

Tools and safety#

• Narrow tools with typed arguments, never a general database or API tool.
• Every state-changing tool takes an idempotency key derived from the conversation and intent.
• Consequential actions return a pending proposal that a human confirms before execution.
• Customer-provided text is untrusted input and is never allowed to authorise an action.

All four of those live in the tool, not in the prompt. A prompt instruction is a request; a tool that will not execute above a limit is a control. The version below is short enough to write on a whiteboard, and writing it is a considerably stronger answer than describing it.

import hashlib, json

REFUND_LEDGER = {}                     # idempotency key -> result
AUDIT = []
AUTO_REFUND_LIMIT_CENTS = 2500

def idempotency_key(conversation_id, intent):
    blob = json.dumps([conversation_id, intent], sort_keys=True)
    return hashlib.sha256(blob.encode()).hexdigest()[:16]

def refund(conversation_id, order_id, amount_cents, approved_by=None):
    """The only way money leaves. Narrow arguments, one job, no free text."""
    key = idempotency_key(conversation_id,
                          {"op": "refund", "order": order_id, "amount": amount_cents})
    if key in REFUND_LEDGER:
        AUDIT.append(("refund.replayed", key))
        return REFUND_LEDGER[key]                  # a retry returns the first result
    if amount_cents > AUTO_REFUND_LIMIT_CENTS and approved_by is None:
        AUDIT.append(("refund.proposed", key, amount_cents))
        return {"status": "awaiting_approval", "key": key, "amount_cents": amount_cents}
    result = {"status": "refunded", "key": key, "amount_cents": amount_cents}
    REFUND_LEDGER[key] = result
    AUDIT.append(("refund.executed", key, amount_cents, approved_by))
    return result

print(refund("conv-91", "ord-4", 1200))
# {'status': 'refunded', 'key': '1018873cf74f3c69', 'amount_cents': 1200}
print(refund("conv-91", "ord-4", 1200))            # the model retried after a timeout
# {'status': 'refunded', 'key': '1018873cf74f3c69', 'amount_cents': 1200}
print(refund("conv-91", "ord-7", 9900))
# {'status': 'awaiting_approval', 'key': '6bb99076f91b1130', 'amount_cents': 9900}
print(refund("conv-91", "ord-7", 9900, approved_by="alex"))
# {'status': 'refunded', 'key': '6bb99076f91b1130', 'amount_cents': 9900}
print(len(REFUND_LEDGER))                          # 2

What goes wrong#

Walk the failures in order and give each one the mechanism that answers it, because a list of risks with no mechanism is the answer an interviewer has heard all day.

The model calls refund twice        idempotency key, second call is a replay
A tool times out, outcome unknown   same key on retry; the ledger decides
The process crashes mid-run         checkpoint per turn, resume not restart
The customer argues it into a
  refund it should not give         the limit is in the tool, not the prompt
The customer pastes instructions
  into the message                  customer text is data in a delimited block;
                                    no tool call is authorised by it
The model invents an order id       the tool looks it up and returns not found,
                                    which is a normal result, not an exception
The provider is down                the agent stops and hands to a human queue;
                                    never degrade into acting without the model
A refund is disputed later          the audit log has the arguments, the result
                                    and the approver

State and audit#

Checkpoint after every turn so a crash resumes rather than repeats. Record every tool call with its arguments, result and who approved it. When a customer disputes a refund, the audit log is the answer.

Be precise about what a checkpoint holds, because it is the difference between resuming and repeating. Store the message history, the pending idempotency keys and the last committed tool result, and write the intent before the tool call rather than the outcome after it. A record written only afterwards is exactly the case you cannot recover from: the process died between the refund and the record, so the replay finds no key to match and issues a second one. Write the intent, execute, then write the outcome. This is an ordinary write-ahead log, and calling it that in the room saves five minutes of explanation.

The audit log is a separate thing from the trace, and conflating them is common. A trace exists for you, can be sampled, and can be thrown away after a month. The audit log exists for a dispute and for whoever asks, a year later, why this account was refunded twice, so it is append-only, it is kept, and it records the approver. If asked whether it should include the model's reasoning, the honest answer is that reasoning text is not evidence of anything and the arguments passed to the tool are.

Autonomy is a ladder, and saying so is worth a mark on its own. Start with the agent drafting every consequential action for a person, which also produces the labelled data you will need. Move one category to automatic when its harm rate over a real sample sits at zero and the volume is large enough for that to mean something. Keep the ability to move it back, and make that a configuration change rather than a deployment, because the day you need it you will need it within the hour.

Cost, out loud#

Per conversation at $1 per million in, $5 per million out:
  19,800 in  x $1 / 1M  =  $0.0198
     900 out x $5 / 1M  =  $0.0045
                           $0.0243, so $24.30 per thousand
  26,700 a day          =  about $650 a day, $19,400 a month

Caching the 1,800-token prefix at a tenth of input price:
  10,800 cached x $0.10 / 1M  =  $0.0011
   9,000 fresh  x $1    / 1M  =  $0.0090
     900 out    x $5    / 1M  =  $0.0045
                                 $0.0146, so $14.60 per thousand
                              =  about $390 a day, $11,700 a month

Then give the number that decides whether the project happens. A support contact handled by a person costs a few pounds; this one costs a penny and a half. Even at a 60 per cent deflection rate the saving is two orders of magnitude larger than the model bill, which means cost per conversation is not the constraint and you should stop optimising it. The constraint is the error rate, because one wrong refund undoes a great many correct ones and a bad enough error undoes the programme. Saying that reframes the whole cost conversation, and it is true.

Evaluating it#

Assert on the end state of a test account, not on what the agent said. Did the refund happen, for the right amount, exactly once? Include adversarial cases where the customer message tries to talk the agent into an unauthorised action.

Concretely: a fixture that seeds a test account with known orders, a list of conversations to replay, and an assertion per case on the ledger afterwards. The assertions that catch real bugs are the negative ones. No refund was issued on a case where none was due. Exactly one row exists where the transcript shows two attempts. Nothing was executed without an approver on any case above the limit. A test suite that only checks the happy path passes on an agent that refunds everybody.

Score it as three numbers rather than one, because they trade against each other and hiding that is how an agent ships with a quiet problem. Resolution rate is how often it finished without a human. Harm rate is how often it did something it should not have, and it is the one with a target of zero rather than a target of high. Escalation quality is whether the ones it handed over were the right ones, since an agent with no harms that escalates everything has simply not been switched on.

What you would cut#

Cut address changes from version one; they are consequential, rare and easily left with humans. Cut the approval user interface and route proposals into the existing support queue as a specially formatted ticket, which is a day of work rather than a fortnight. Cut multi-turn state by resolving one issue per conversation. Do not cut the audit log or the idempotency key: both are cheap to build now and effectively impossible to add later, because retrofitting them means reconciling a history of actions nobody recorded.