Designing for a component that is sometimes wrong

An AI feature fails differently from ordinary software. It does not throw an error, it returns something plausible and wrong, which the user has no obvious way to catch. This section is about designing for that: showing the working so an answer can be checked, making a correction cheap rather than a restart, and setting expectations so confidence is not mistaken for accuracy.

The failure mode is plausibility#

Ordinary software fails visibly. It throws, it shows an error, it does nothing. An AI feature fails by producing something confident, well written and wrong. The user has no signal that anything went awry.

That changes the design problem. You are not trying to eliminate mistakes, which is not achievable. You are designing so that mistakes are visible, cheap to correct and rarely consequential.

Choosing what not to build with a model#

The first product decision is refusal, and in an interview it is the one that separates people. Handed a wish list, most candidates start designing. The stronger move is to sort it, out loud, into what a model should do, what it should draft for someone else to do, and what it should not touch. Four questions do the sorting.

Is there a right answer?          no  -> a model is fine, taste is the point
                                  yes -> keep going

Can the user tell it is wrong?    no  -> do not ship it as an answer
                                  yes -> keep going

Is the action reversible?         no  -> a person approves, always
                                  yes -> keep going

Does an existing method work?     yes -> use it; a regex that is always
                                         right beats a model that is usually

The second question is the sharp one. A summary of a document the user has not read is unverifiable by the person receiving it, which is exactly the case where a wrong answer travels furthest. A summary of a document sitting open next to it is verifiable in seconds. Same model, same prompt, same accuracy, and only one of them is a product. When someone in a product round asks how you would make a feature trustworthy, that distinction is the answer they are listening for, not a better prompt.

Where a probabilistic component is acceptable#

There is a number underneath this and it is worth being able to produce it. The feature is worth having when the time it saves exceeds the time it costs, and the time it costs has three parts: checking every answer, redoing the ones the check catches, and cleaning up after the ones it does not.

MANUAL   = 12.0    # minutes to do the task by hand
VERIFY   = 2.5     # minutes to check the model's answer
REWORK   = 1.0     # extra minutes lost when you checked, found it wrong, and redid it
INCIDENT = 240.0   # minutes a wrong answer costs once it has escaped downstream

def minutes(accuracy, caught):
    wrong = 1 - accuracy
    return (VERIFY
            + wrong * caught * (MANUAL + REWORK)
            + wrong * (1 - caught) * INCIDENT)

def breakeven(caught):
    lo, hi = 0.0, 1.0
    for _ in range(60):
        mid = (lo + hi) / 2
        lo, hi = (lo, mid) if minutes(mid, caught) < MANUAL else (mid, hi)
    return hi

print(f"by hand: {MANUAL:.1f} min a task, always right\n")
print(f"{'accuracy':>9}{'with the model':>17}{'vs by hand':>13}")
for acc in (0.995, 0.99, 0.97, 0.95, 0.90, 0.80):
    m = minutes(acc, 0.85)
    print(f"{acc:>9.1%}{m:>14.2f} min{(MANUAL - m)/MANUAL:>12.0%}")

print(f"\ncheck catches 85% of errors -> break-even accuracy {breakeven(0.85):.1%}")
print(f"check catches 99% of errors -> break-even accuracy {breakeven(0.99):.1%}")

# by hand: 12.0 min a task, always right
#
#  accuracy   with the model   vs by hand
#     99.5%          2.74 min         77%
#     99.0%          2.97 min         75%
#     97.0%          3.91 min         67%
#     95.0%          4.85 min         60%
#     90.0%          7.20 min         40%
#     80.0%         11.91 min          1%
#
# check catches 85% of errors -> break-even accuracy 79.8%
# check catches 99% of errors -> break-even accuracy 37.8%

The two break-even figures are the finding. With a check that catches 85 per cent of errors the model has to be 79.8 per cent accurate before the feature is worth having. Improve the check to 99 per cent and the bar falls to 37.8 per cent. Improving the check moved the requirement further than any plausible improvement to the model would have, and improving the check is engineering you control, on a schedule you can commit to, unlike waiting for a better model.

It also tells you when to refuse. The whole of that 79.8 per cent is the 15 per cent of errors that escape, at four hours each. Raise the cost of an escaped error — a wrong figure in a filing, a wrong dosage, a wrong name on a legal document — and the break-even accuracy goes above anything achievable, and the honest conclusion is that this task does not get a probabilistic component at all, or gets one only behind an approval step. That is a defensible position in a product round, and it is much stronger than promising to evaluate carefully.

Show the work#

• Citations, linked to the actual source, so a user can check in one click.
• Show what was retrieved or considered, not only the conclusion.
• Say when confidence is low rather than presenting everything with the same certainty.
• Make it obvious the output came from a model, so it is read with appropriate scepticism.

Correction should be one gesture#

Users will need to fix things. Editing in place beats regenerating, undo beats an apology, and a specific correction that sticks beats rerunning the whole thing and hoping.

Weak:    a wall of generated text, with a Regenerate button

Better:  editable fields, each showing where the value came from,
         with a Looks wrong control next to each one

Field-level output is doing more than one job there. It makes the correction local, so the user keeps the nine values that were right. It gives you per-field accuracy data for free, because you now know which field was edited rather than only that the output was rejected. And it lets the interface show provenance next to each value, which is what makes the check quick. A single block of prose supports none of those, which is why the first design decision on an extraction feature is usually its granularity, not its prompt.

Set expectations in the interface#

Overclaiming in the copy is how trust is lost. Calling something a draft rather than an answer changes how carefully it is read, and it is honest.

• Name it accurately: draft, suggestion, summary. Not answer, unless it is one.
• Say what it looked at, so the user knows the scope of the claim.
• Say when it does not know rather than producing something to fill the space.
• Do not use confidence percentages unless they are calibrated, because an uncalibrated number is worse than none.

Match autonomy to the cost of being wrong#

Suggest        user decides everything      reversible, low stakes
Draft          user reviews and edits       moderate stakes
Act on approval user confirms each action   high stakes
Act            fully autonomous             only when errors are cheap
                                            and detectable

The level is a property of the action, not of the feature, so one feature usually sits at several. An agent that reads a mailbox, drafts three replies, files two documents and issues a refund is at Act for reading, Draft for the replies, Act for the filing because it is reversible in one click, and Act on approval for the refund because it is not. Picking one level for the whole feature is how products end up either useless or frightening, and naming the four actions separately is a quick way to sound like someone who has shipped one.

What to skip#

Skip the confidence score; getting it calibrated is real work and an uncalibrated one actively misleads. Skip the disclaimer banner, which everybody stops seeing in a day and which is usually there to protect the team rather than the user — put the signal next to the value instead. And skip designing the happy path first: the wrong-answer path is the one that determines whether anybody keeps using this, so draw that screen before you draw the other one.