Knowing whether the AI feature actually helps

An eval tells you the model is doing what you asked of it. It does not tell you whether users are better off, and those two things come apart more often than anyone expects. This section is about the signals that answer the second question, which are what people do with the output rather than what they say about it, and about the fact that most of those signals only exist if the product was built to record them. It ends on the decision underneath all of it: which tasks should be automated at all, and which should produce a draft a person signs off.

Thumbs up and down#

Explicit ratings are useful and heavily biased. Few users click them, and those who do are often unusually pleased or annoyed. Use them to find cases to read, not as your quality number.

Put a number on the bias before you quote a rating to anyone. If one user in fifty rates, an 85 per cent thumbs-up rate is a statement about two per cent of your traffic, selected for having felt strongly. It is a sampling frame, not a measurement. The one thing it is good for is triage: sort by thumbs-down, read twenty, and you will have the three failure modes worth building evals around by lunchtime.

What people do tells you more#

• Edits: if users rewrite most of a draft before sending, the draft was not good. Measure how much of the text survives.
• Retries and rephrasing: asking the same thing again means the first answer failed.
• Copy, accept, apply: the user took the output and used it.
• Abandonment: the user left mid-task or switched to the old way of doing it.
• Escalation: the user asked for a person.

These signals only exist if the product is built to capture them. Log draft and final versions, accept and reject actions, and conversation outcomes from the start.

Edit survival is the most useful of them and the least often implemented, because it needs both versions kept. It is a diff, and it separates the draft that needed a word changed from the draft that was thrown away and rewritten — two things that are the same event in every accept-rate dashboard.

import difflib

def survival(draft, sent):
    """Share of the draft's characters that made it into what the user sent."""
    m = difflib.SequenceMatcher(None, draft, sent)
    kept = sum(block.size for block in m.get_matching_blocks())
    return kept / len(draft)

pairs = [
    ("Thanks for getting in touch. I have refunded the order and you should see "
     "the money within 3-5 working days.",
     "Thanks for getting in touch. I have refunded order 88120 and you should see "
     "the money within 3-5 working days."),
    ("Dear valued customer, we sincerely apologise for the inconvenience caused "
     "by this issue and appreciate your patience.",
     "Sorry about that. Sending a replacement today."),
    ("Your subscription renews on 1 March at the current rate.",
     "Your subscription renews on 1 March at the current rate."),
]
for draft, sent in pairs:
    print(f"{survival(draft, sent):>6.0%} of the draft survived   {sent[:44]}...")

avg = sum(survival(d, s) for d, s in pairs) / len(pairs)
print(f"\nmean survival {avg:.0%} over {len(pairs)} messages")

#    96% of the draft survived   Thanks for getting in touch. I have refunded...
#    17% of the draft survived   Sorry about that. Sending a replacement toda...
#   100% of the draft survived   Your subscription renews on 1 March at the c...
#
# mean survival 71% over 3 messages

The mean is the trap in that output. Seventy-one per cent across three messages describes none of them: two were essentially untouched and one was discarded. Report the distribution, or a share below a threshold — the proportion of drafts where under half survived is a better single number than any average, because it counts the drafts that wasted the user's time rather than averaging them away against the good ones.

Tie it to an outcome#

In the end the question is whether a business outcome moved: tickets resolved without a person, time to complete a task, conversion. Agree on one before launch, with guardrails such as complaint rate and cost that must not get worse.

Then work out, before you promise anyone an answer, whether you have the traffic to detect the change you are hoping for. This is the arithmetic that turns a confident plan into an honest one, and it takes a minute.

import math

Z_ALPHA, Z_BETA = 1.96, 0.84      # two-sided 5% significance, 80% power

def per_arm(p0, lift):
    p1 = p0 + lift
    pbar = (p0 + p1) / 2
    num = (Z_ALPHA * math.sqrt(2 * pbar * (1 - pbar))
           + Z_BETA * math.sqrt(p0 * (1 - p0) + p1 * (1 - p1))) ** 2
    return math.ceil(num / lift ** 2)

BASE = 0.62            # tickets resolved without a person today
print(f"{'lift you want to detect':>24}{'users per arm':>16}{'days at 400/day/arm':>22}")
for lift in (0.10, 0.05, 0.03, 0.02, 0.01):
    n = per_arm(BASE, lift)
    print(f"{lift:>23.0%}{n:>16,}{n/400:>21.0f}")

print(f"\nA 2-point improvement on a {BASE:.0%} base needs {per_arm(BASE, 0.02):,} "
      f"users in each arm.")

#  lift you want to detect   users per arm   days at 400/day/arm
#                     10%             346                    1
#                      5%           1,435                    4
#                      3%           4,037                   10
#                      2%           9,137                   23
#                      1%          36,749                   92
#
# A 2-point improvement on a 62% base needs 9,137 users in each arm.

That table decides what you can honestly claim. A ten-point jump shows up in a day. The two-point improvement that most AI features actually deliver needs over nine thousand users per arm, which at a thousand users a week is months, by which time you will have changed the prompt four times and invalidated the experiment. Saying that out loud, and then proposing a faster proxy — edit survival, escalation rate, time to complete — is a stronger answer in a product round than designing an A/B test that would never have finished.

Handing off to a person#

Every AI feature that talks to customers needs a way out. A good handoff passes the conversation, what the assistant tried, and what it thinks the problem is, so the customer does not repeat themselves. A bad one is a dead end or a loop.

What to automate#

              cheap to be wrong        costly to be wrong
frequent      automate                  draft, person approves
rare          automate or skip          leave to people

The product round, as it is actually run#

It is forty-five minutes, it opens with a sentence rather than a specification, and the opening sentence is deliberately underspecified. Something like: our support team is drowning, what would you build with an LLM. Nobody is looking for a feature list. They are watching whether you narrow before you design.

0-5    ask who the user is, what they do today, and how you would
       know it worked. Do not design yet.
5-10   pick one user and one task, out loud, and say what you are
       leaving out and why
10-20  the shape: what the model does, what it does not touch,
       where a person sits, what the interface promises
20-30  being wrong: the failure modes, which are visible to the
       user, and what it costs to correct each one
30-40  measurement: the one outcome metric, the guardrails, the
       behavioural signals, and what you have to log to get them
40-45  what you would cut, what you would not build at all, and
       what would make you kill it

The rubric behind it is shorter than the sequence. Did you narrow to one user and one task rather than designing a platform. Did you say what you would not build, and give a reason. Did you design for the model being wrong instead of assuming accuracy improves. Did you name a metric you could actually collect, and say what you would have to log. Did you make a decision when the interviewer pushed back, rather than adding both options. Answers that fail here usually fail on the same two things: a feature list with no user, and an accuracy assumption never stated.

Two moves reliably raise the grade. Give a number, any defensible number — a volume, a cost per call, a break-even accuracy — because a product round on AI is an engineering conversation and the arithmetic in this topic is what makes it one. And answer the being-wrong question with a screen rather than a policy: describe what the user sees when the model gets it wrong, and who finds out.

What to skip#

Skip the dashboard nobody reads. Three numbers reviewed weekly beat thirty that are never opened. Skip A/B testing a change you would ship regardless of the result, which is most prompt changes; ship it behind a flag and watch the guardrails. And skip proving value from ratings alone — if the only number you have at the end of a quarter is a thumbs-up rate, the thing that went wrong happened at the start, when nobody logged the drafts.