Product judgement for AI

An AI feature fails differently from ordinary software. It does not throw an error, it returns something plausible and wrong, which the user has no obvious way to catch. This section is about designing for that: showing the working so an answer can be checked, making a correction cheap rather than a restart, and setting expectations so confidence is not mistaken for accuracy.

The failure mode is plausibility#

Ordinary software fails visibly. It throws, it shows an error, it does nothing. An AI feature fails by producing something confident, well written and wrong. The user has no signal that anything went awry.

That changes the design problem. You are not trying to eliminate mistakes, which is not achievable. You are designing so that mistakes are visible, cheap to correct and rarely consequential.

Choosing what not to build with a model#

The first product decision is refusal, and in an interview it is the one that separates people. Handed a wish list, most candidates start designing. The stronger move is to sort it, out loud, into what a model should do, what it should draft for someone else to do, and what it should not touch. Four questions do the sorting.

Is there a right answer?          no  -> a model is fine, taste is the point
                                  yes -> keep going

Can the user tell it is wrong?    no  -> do not ship it as an answer
                                  yes -> keep going

Is the action reversible?         no  -> a person approves, always
                                  yes -> keep going

Does an existing method work?     yes -> use it; a regex that is always
                                         right beats a model that is usually

The second question is the sharp one. A summary of a document the user has not read is unverifiable by the person receiving it, which is exactly the case where a wrong answer travels furthest. A summary of a document sitting open next to it is verifiable in seconds. Same model, same prompt, same accuracy, and only one of them is a product. When someone in a product round asks how you would make a feature trustworthy, that distinction is the answer they are listening for, not a better prompt.

Where a probabilistic component is acceptable#

There is a number underneath this and it is worth being able to produce it. The feature is worth having when the time it saves exceeds the time it costs, and the time it costs has three parts: checking every answer, redoing the ones the check catches, and cleaning up after the ones it does not.

MANUAL   = 12.0    # minutes to do the task by hand
VERIFY   = 2.5     # minutes to check the model's answer
REWORK   = 1.0     # extra minutes lost when you checked, found it wrong, and redid it
INCIDENT = 240.0   # minutes a wrong answer costs once it has escaped downstream

def minutes(accuracy, caught):
    wrong = 1 - accuracy
    return (VERIFY
            + wrong * caught * (MANUAL + REWORK)
            + wrong * (1 - caught) * INCIDENT)

def breakeven(caught):
    lo, hi = 0.0, 1.0
    for _ in range(60):
        mid = (lo + hi) / 2
        lo, hi = (lo, mid) if minutes(mid, caught) < MANUAL else (mid, hi)
    return hi

print(f"by hand: {MANUAL:.1f} min a task, always right\n")
print(f"{'accuracy':>9}{'with the model':>17}{'vs by hand':>13}")
for acc in (0.995, 0.99, 0.97, 0.95, 0.90, 0.80):
    m = minutes(acc, 0.85)
    print(f"{acc:>9.1%}{m:>14.2f} min{(MANUAL - m)/MANUAL:>12.0%}")

print(f"\ncheck catches 85% of errors -> break-even accuracy {breakeven(0.85):.1%}")
print(f"check catches 99% of errors -> break-even accuracy {breakeven(0.99):.1%}")

# by hand: 12.0 min a task, always right
#
#  accuracy   with the model   vs by hand
#     99.5%          2.74 min         77%
#     99.0%          2.97 min         75%
#     97.0%          3.91 min         67%
#     95.0%          4.85 min         60%
#     90.0%          7.20 min         40%
#     80.0%         11.91 min          1%
#
# check catches 85% of errors -> break-even accuracy 79.8%
# check catches 99% of errors -> break-even accuracy 37.8%

The two break-even figures are the finding. With a check that catches 85 per cent of errors the model has to be 79.8 per cent accurate before the feature is worth having. Improve the check to 99 per cent and the bar falls to 37.8 per cent. Improving the check moved the requirement further than any plausible improvement to the model would have, and improving the check is engineering you control, on a schedule you can commit to, unlike waiting for a better model.

It also tells you when to refuse. The whole of that 79.8 per cent is the 15 per cent of errors that escape, at four hours each. Raise the cost of an escaped error — a wrong figure in a filing, a wrong dosage, a wrong name on a legal document — and the break-even accuracy goes above anything achievable, and the honest conclusion is that this task does not get a probabilistic component at all, or gets one only behind an approval step. That is a defensible position in a product round, and it is much stronger than promising to evaluate carefully.

Show the work#

• Citations, linked to the actual source, so a user can check in one click.
• Show what was retrieved or considered, not only the conclusion.
• Say when confidence is low rather than presenting everything with the same certainty.
• Make it obvious the output came from a model, so it is read with appropriate scepticism.

Correction should be one gesture#

Users will need to fix things. Editing in place beats regenerating, undo beats an apology, and a specific correction that sticks beats rerunning the whole thing and hoping.

Weak:    a wall of generated text, with a Regenerate button

Better:  editable fields, each showing where the value came from,
         with a Looks wrong control next to each one

Field-level output is doing more than one job there. It makes the correction local, so the user keeps the nine values that were right. It gives you per-field accuracy data for free, because you now know which field was edited rather than only that the output was rejected. And it lets the interface show provenance next to each value, which is what makes the check quick. A single block of prose supports none of those, which is why the first design decision on an extraction feature is usually its granularity, not its prompt.

Set expectations in the interface#

Overclaiming in the copy is how trust is lost. Calling something a draft rather than an answer changes how carefully it is read, and it is honest.

• Name it accurately: draft, suggestion, summary. Not answer, unless it is one.
• Say what it looked at, so the user knows the scope of the claim.
• Say when it does not know rather than producing something to fill the space.
• Do not use confidence percentages unless they are calibrated, because an uncalibrated number is worse than none.

Match autonomy to the cost of being wrong#

Suggest        user decides everything      reversible, low stakes
Draft          user reviews and edits       moderate stakes
Act on approval user confirms each action   high stakes
Act            fully autonomous             only when errors are cheap
                                            and detectable

The level is a property of the action, not of the feature, so one feature usually sits at several. An agent that reads a mailbox, drafts three replies, files two documents and issues a refund is at Act for reading, Draft for the replies, Act for the filing because it is reversible in one click, and Act on approval for the refund because it is not. Picking one level for the whole feature is how products end up either useless or frightening, and naming the four actions separately is a quick way to sound like someone who has shipped one.

What to skip#

Skip the confidence score; getting it calibrated is real work and an uncalibrated one actively misleads. Skip the disclaimer banner, which everybody stops seeing in a day and which is usually there to protect the team rather than the user — put the signal next to the value instead. And skip designing the happy path first: the wrong-answer path is the one that determines whether anybody keeps using this, so draw that screen before you draw the other one.

An eval tells you the model is doing what you asked of it. It does not tell you whether users are better off, and those two things come apart more often than anyone expects. This section is about the signals that answer the second question, which are what people do with the output rather than what they say about it, and about the fact that most of those signals only exist if the product was built to record them. It ends on the decision underneath all of it: which tasks should be automated at all, and which should produce a draft a person signs off.

Thumbs up and down#

Explicit ratings are useful and heavily biased. Few users click them, and those who do are often unusually pleased or annoyed. Use them to find cases to read, not as your quality number.

Put a number on the bias before you quote a rating to anyone. If one user in fifty rates, an 85 per cent thumbs-up rate is a statement about two per cent of your traffic, selected for having felt strongly. It is a sampling frame, not a measurement. The one thing it is good for is triage: sort by thumbs-down, read twenty, and you will have the three failure modes worth building evals around by lunchtime.

What people do tells you more#

• Edits: if users rewrite most of a draft before sending, the draft was not good. Measure how much of the text survives.
• Retries and rephrasing: asking the same thing again means the first answer failed.
• Copy, accept, apply: the user took the output and used it.
• Abandonment: the user left mid-task or switched to the old way of doing it.
• Escalation: the user asked for a person.

These signals only exist if the product is built to capture them. Log draft and final versions, accept and reject actions, and conversation outcomes from the start.

Edit survival is the most useful of them and the least often implemented, because it needs both versions kept. It is a diff, and it separates the draft that needed a word changed from the draft that was thrown away and rewritten — two things that are the same event in every accept-rate dashboard.

import difflib

def survival(draft, sent):
    """Share of the draft's characters that made it into what the user sent."""
    m = difflib.SequenceMatcher(None, draft, sent)
    kept = sum(block.size for block in m.get_matching_blocks())
    return kept / len(draft)

pairs = [
    ("Thanks for getting in touch. I have refunded the order and you should see "
     "the money within 3-5 working days.",
     "Thanks for getting in touch. I have refunded order 88120 and you should see "
     "the money within 3-5 working days."),
    ("Dear valued customer, we sincerely apologise for the inconvenience caused "
     "by this issue and appreciate your patience.",
     "Sorry about that. Sending a replacement today."),
    ("Your subscription renews on 1 March at the current rate.",
     "Your subscription renews on 1 March at the current rate."),
]
for draft, sent in pairs:
    print(f"{survival(draft, sent):>6.0%} of the draft survived   {sent[:44]}...")

avg = sum(survival(d, s) for d, s in pairs) / len(pairs)
print(f"\nmean survival {avg:.0%} over {len(pairs)} messages")

#    96% of the draft survived   Thanks for getting in touch. I have refunded...
#    17% of the draft survived   Sorry about that. Sending a replacement toda...
#   100% of the draft survived   Your subscription renews on 1 March at the c...
#
# mean survival 71% over 3 messages

The mean is the trap in that output. Seventy-one per cent across three messages describes none of them: two were essentially untouched and one was discarded. Report the distribution, or a share below a threshold — the proportion of drafts where under half survived is a better single number than any average, because it counts the drafts that wasted the user's time rather than averaging them away against the good ones.

Tie it to an outcome#

In the end the question is whether a business outcome moved: tickets resolved without a person, time to complete a task, conversion. Agree on one before launch, with guardrails such as complaint rate and cost that must not get worse.

Then work out, before you promise anyone an answer, whether you have the traffic to detect the change you are hoping for. This is the arithmetic that turns a confident plan into an honest one, and it takes a minute.

import math

Z_ALPHA, Z_BETA = 1.96, 0.84      # two-sided 5% significance, 80% power

def per_arm(p0, lift):
    p1 = p0 + lift
    pbar = (p0 + p1) / 2
    num = (Z_ALPHA * math.sqrt(2 * pbar * (1 - pbar))
           + Z_BETA * math.sqrt(p0 * (1 - p0) + p1 * (1 - p1))) ** 2
    return math.ceil(num / lift ** 2)

BASE = 0.62            # tickets resolved without a person today
print(f"{'lift you want to detect':>24}{'users per arm':>16}{'days at 400/day/arm':>22}")
for lift in (0.10, 0.05, 0.03, 0.02, 0.01):
    n = per_arm(BASE, lift)
    print(f"{lift:>23.0%}{n:>16,}{n/400:>21.0f}")

print(f"\nA 2-point improvement on a {BASE:.0%} base needs {per_arm(BASE, 0.02):,} "
      f"users in each arm.")

#  lift you want to detect   users per arm   days at 400/day/arm
#                     10%             346                    1
#                      5%           1,435                    4
#                      3%           4,037                   10
#                      2%           9,137                   23
#                      1%          36,749                   92
#
# A 2-point improvement on a 62% base needs 9,137 users in each arm.

That table decides what you can honestly claim. A ten-point jump shows up in a day. The two-point improvement that most AI features actually deliver needs over nine thousand users per arm, which at a thousand users a week is months, by which time you will have changed the prompt four times and invalidated the experiment. Saying that out loud, and then proposing a faster proxy — edit survival, escalation rate, time to complete — is a stronger answer in a product round than designing an A/B test that would never have finished.

Handing off to a person#

Every AI feature that talks to customers needs a way out. A good handoff passes the conversation, what the assistant tried, and what it thinks the problem is, so the customer does not repeat themselves. A bad one is a dead end or a loop.

What to automate#

              cheap to be wrong        costly to be wrong
frequent      automate                  draft, person approves
rare          automate or skip          leave to people

The product round, as it is actually run#

It is forty-five minutes, it opens with a sentence rather than a specification, and the opening sentence is deliberately underspecified. Something like: our support team is drowning, what would you build with an LLM. Nobody is looking for a feature list. They are watching whether you narrow before you design.

0-5    ask who the user is, what they do today, and how you would
       know it worked. Do not design yet.
5-10   pick one user and one task, out loud, and say what you are
       leaving out and why
10-20  the shape: what the model does, what it does not touch,
       where a person sits, what the interface promises
20-30  being wrong: the failure modes, which are visible to the
       user, and what it costs to correct each one
30-40  measurement: the one outcome metric, the guardrails, the
       behavioural signals, and what you have to log to get them
40-45  what you would cut, what you would not build at all, and
       what would make you kill it

The rubric behind it is shorter than the sequence. Did you narrow to one user and one task rather than designing a platform. Did you say what you would not build, and give a reason. Did you design for the model being wrong instead of assuming accuracy improves. Did you name a metric you could actually collect, and say what you would have to log. Did you make a decision when the interviewer pushed back, rather than adding both options. Answers that fail here usually fail on the same two things: a feature list with no user, and an accuracy assumption never stated.

Two moves reliably raise the grade. Give a number, any defensible number — a volume, a cost per call, a break-even accuracy — because a product round on AI is an engineering conversation and the arithmetic in this topic is what makes it one. And answer the being-wrong question with a screen rather than a policy: describe what the user sees when the model gets it wrong, and who finds out.

What to skip#

Skip the dashboard nobody reads. Three numbers reviewed weekly beat thirty that are never opened. Skip A/B testing a change you would ship regardless of the result, which is most prompt changes; ship it behind a flag and watch the guardrails. And skip proving value from ratings alone — if the only number you have at the end of a quarter is a thumbs-up rate, the thing that went wrong happened at the start, when nobody logged the drafts.