Reasoning, and improving a prompt honestly

Two things here, and they belong together. One is the family of techniques for making a model work a problem through in stages instead of answering at once, several of which have aged badly and now make newer models worse. The other is how you would ever know: a small fixed set of cases, so that editing a prompt produces a number rather than a feeling. Without that second half, prompt work is fiddling until the one example you keep re-reading looks right.

What actually moves the number#

Prompt advice is mostly a list of tricks with no ordering, which is useless when you have an afternoon and a prompt that fails one time in five. This is the order worth working in, roughly by how much it changes the result per hour spent.

change                                 why it works
say exactly what the output must be    removes the largest source of failure,
                                       which is a good answer you cannot parse
define the boundary between labels     most errors sit on one or two boundaries,
                                       not spread evenly over the task
give the model an escape hatch         without "unknown" it must guess, and it will
put the long document in delimiters    so instructions cannot be read as content
show two or three examples             for anything easier to demonstrate than describe,
                                       including the hard case and the empty one
split a prompt doing four jobs         each job gets easier, and failures become locatable
ask for reasoning before the answer    on older or smaller models, on multi-step tasks
                                       only; measure it, it can now hurt
add flattery, urgency, a tip           nothing measurable

Output format is the cheapest win, and you can measure it#

Almost every prompt in production is read by code rather than by a person, so an answer that is right and unparseable is a failure. This one you can quantify without calling a model at all: collect the replies you already have and run your parser over them. The twelve below are the shapes a model actually returns when asked for JSON.

import json, re

FENCE = chr(96) * 3          # the code fence models like to wrap JSON in

replies = [
    '{"label": "refund", "confidence": 0.9}',
    'Sure! Here is the JSON:\n{"label": "refund", "confidence": 0.9}',
    FENCE + 'json\n{"label": "refund", "confidence": 0.9}\n' + FENCE,
    FENCE + '\n{"label": "refund", "confidence": 0.9}\n' + FENCE,
    '{"label": "refund", "confidence": 0.9,}',
    "{'label': 'refund', 'confidence': 0.9}",
    '{"label": "refund", "confidence": None}',
    '{"label": "refund", "confidence": 0.9}\nLet me know if you need anything else.',
    '  {"label": "refund", "confidence": 0.9}  ',
    '{\n  "label": "refund",\n  "confidence": 0.9\n}',
    'The label is refund.',
    '{"label": "refund"}',
]

def strict(text):
    return json.loads(text)

def tolerant(text):
    text = re.sub(r"^[^{]*", "", text, count=1)      # drop any preamble
    text = re.sub(r"[^}]*$", "", text, count=1)      # drop any trailing chat
    text = text.replace("'", '"').replace("None", "null")
    text = re.sub(r",\s*}", "}", text)              # trailing comma
    return json.loads(text)

def validated(text):
    obj = tolerant(text)
    if obj.get("label") not in {"refund", "billing", "other"}:
        raise ValueError("label not in the allowed set")
    if not isinstance(obj.get("confidence"), float):
        raise ValueError("confidence missing or not a number")
    return obj

for name, parse in (("json.loads", strict), ("tolerant", tolerant),
                    ("validated", validated)):
    ok = 0
    for r in replies:
        try:
            parse(r)
            ok += 1
        except Exception:
            pass
    print("%-10s parsed %2d of %d" % (name, ok, len(replies)))

# json.loads parsed  4 of 12
# tolerant   parsed 11 of 12
# validated  parsed  9 of 12

Read those three numbers as three different afternoons. If your pipeline uses the first parser, a third of your failures are formatting rather than judgement, and no amount of rewording the task will fix them. Tolerant parsing is worth having whatever else you do, because the model will wrap JSON in a fence eventually. But notice what tolerance costs: it accepted a reply with a null confidence and one with the field missing, either of which becomes a confident wrong decision downstream unless something checks the shape.

The prompt-side fixes are the same three lines every time. State the exact keys and their types, give the allowed values as a closed list, and say what to emit when the answer is none of them. Where the provider supports a structured output or tool schema, use it and delete the tolerant parser, since a schema the server enforces beats a regular expression that guesses.

Ordering, and where the instruction goes#

Position is not neutral. The reliable rules are the ones that follow from how the input is read rather than from folklore: put long documents in explicit delimiters so nothing inside them can be mistaken for an instruction, and put the actual question where it cannot be buried. With a long document, that means stating the task before it and restating it after, which costs a dozen tokens and removes a whole class of failure where the model summarises a document you wanted classified.

Two orderings genuinely matter and both are cheap to check. In few-shot prompts the order of examples can shift the output distribution, particularly if all the easy cases come first or the labels arrive in a block. And in a long prompt, whatever sits immediately before the model starts writing has more influence than anything else, so the output contract belongs at the end rather than the top. Neither is a law. Both are two runs of your eval away from being settled for your model and your task.

Chain of thought#

Asking a model to work through its reasoning before answering used to produce a large improvement on multi-step problems. It still helps on older and smaller models, and on genuinely multi-step tasks.

It has become much less necessary. Newer reasoning models already do this internally, and instructing them to think step by step can make results worse by interfering with a process that was working. Test it rather than adding it by reflex.

When you do want it, ask for the reasoning in a place you can throw away, such as a separate field or a tagged block before the answer, rather than mixed into the output your code parses. That keeps the format contract intact, and it lets you look at the reasoning when a case fails, which is worth more than the accuracy change on most tasks.

Split the task#

When a prompt is trying to do four things at once, quality on all four drops. Splitting into separate calls usually beats a longer prompt, and it has a large practical advantage: you can tell which step failed.

One prompt:   read the document, extract the fields, validate
              them, and write a summary

Three calls:  extract  ->  validate  ->  summarise

The second version costs more per document and is far cheaper
to debug, because a failure points at one step.

The other gain is that each step becomes separately measurable. A single prompt gives you one number that mixes four skills, so a change that improves extraction and damages summarising looks like no change at all. Three prompts give you three numbers, and the one that moved tells you what you did.

Iterate against a test set, not against memory#

This is the part people skip, and it is the part that matters. Changing a prompt and eyeballing the next output tells you almost nothing, because you remember the failures you were looking for and not the ones you were not.

1. 1.Collect twenty real inputs, including the ones that currently fail.
2. 2.Write the expected outcome for each, before you touch the prompt.
3. 3.Score the current prompt. That number is your baseline.
4. 4.Change one thing. Score again.
5. 5.Keep the change only if the number moved.

Twenty cases and a script is enough to start. This is the beginning of evals, and doing it at prompt level makes the full discipline feel obvious later. It is also where the next two experiments come from, because the first question anyone should ask about a number is how much it wobbles on its own.

Twenty cases cannot see what you think they can#

Sixteen out of twenty becoming seventeen out of twenty feels like progress. It is worth knowing exactly how often that happens when nothing has changed at all, and the arithmetic is a few lines of standard library.

from math import comb

def at_least(k, n, p):
    """Probability of scoring k or more out of n when the true rate is p."""
    return sum(comb(n, i) * p**i * (1 - p)**(n - i) for i in range(k, n + 1))

print("prompt A scores 16/20. Prompt B scores 17/20. If B is no better:")
print("  P(B scores 17 or more | true rate 0.80) = %.3f" % at_least(17, 20, 0.80))
print("  P(B scores 18 or more | true rate 0.80) = %.3f" % at_least(18, 20, 0.80))
print()
print("How big must the set be before a real 80 -> 90 percent jump is visible?")
for n in (20, 50, 100, 200, 400):
    # the score you would need to be 95% sure the old prompt did not produce it
    need = next(k for k in range(n + 1) if at_least(k, n, 0.80) < 0.05)
    print("  n=%3d: need %3d/%3d (%.0f%%) to clear noise; a true 90%% model "
          "delivers that %.0f%% of the time" % (n, need, n, 100 * need / n,
                                                100 * at_least(need, n, 0.90)))

# prompt A scores 16/20. Prompt B scores 17/20. If B is no better:
#   P(B scores 17 or more | true rate 0.80) = 0.411
#   P(B scores 18 or more | true rate 0.80) = 0.206
#
# How big must the set be before a real 80 -> 90 percent jump is visible?
#   n= 20: need  20/ 20 (100%) to clear noise; a true 90% model delivers that 12% of the time
#   n= 50: need  45/ 50 (90%) to clear noise; a true 90% model delivers that 62% of the time
#   n=100: need  87/100 (87%) to clear noise; a true 90% model delivers that 88% of the time
#   n=200: need 170/200 (85%) to clear noise; a true 90% model delivers that 99% of the time
#   n=400: need 334/400 (84%) to clear noise; a true 90% model delivers that 100% of the time

The second table is the one to keep. At a hundred cases you catch a ten-point improvement most of the time, and that is the number to aim at before claiming anything. Below fifty, you are not measuring the prompt, you are measuring which twenty inputs you happened to collect. This does not mean twenty cases are worthless: they are excellent for catching a prompt that is broken, a format that will not parse, or a change that made things obviously worse. They are simply not an instrument for small improvements.

One cheap trick makes a small set go further. Compare the two prompts only on the cases where they disagree, and count how many each got right. If they differ on three cases and the new prompt wins two of them, nothing has been shown, and you have learned that in the time it takes to read three rows.

Why you have to hold cases back#

The trap at the end of this section is usually stated as advice. It is better stated as a measurement. Below, several prompt variants are scored on the same twenty cases and the best one is kept. Every variant is secretly identical, with a true pass rate of 0.75, so nothing in this experiment can possibly be an improvement.

import numpy as np

rng = np.random.default_rng(0)
trials, n_cases, true_rate = 4000, 20, 0.75
print("variants tried   best dev score   same prompt, fresh cases")
for n_variants in (1, 3, 10, 30):
    dev = rng.random((trials, n_variants, n_cases)) < true_rate
    winner = dev.mean(axis=2).max(axis=1)
    fresh = (rng.random((trials, n_cases)) < true_rate).mean(axis=1)
    print("%14d   %14.3f   %23.3f" % (n_variants, winner.mean(), fresh.mean()))

# variants tried   best dev score   same prompt, fresh cases
#              1            0.751                     0.752
#              3            0.828                     0.749
#             10            0.889                     0.750
#             30            0.927                     0.749

That is fourteen points of pure illusion from an afternoon of honest work, and it is why the number you report must come from cases you did not select on. Split the set before you start: two thirds to iterate against, one third you look at once, at the end, and never again. If the held-out number is far below the one you were watching, you have been tuning to noise, and the experiment above says how much of it to expect from how many variants you tried.

A prompt, before and after#

Everything above, applied to one real prompt. The task is routing a support message to a queue.

BEFORE
  You are an expert customer support agent with 20 years of
  experience. Please carefully analyse the following customer
  message and determine which category it belongs to. This is
  very important, so think step by step. Categories: refund,
  billing, technical, other.

  Message: {message}

AFTER
  Route a support message to one queue.

  <message>
  {message}
  </message>

  Queues, in priority order. Use the first one that applies:
    refund     the customer is asking for money back
    billing    a charge, invoice or subscription question,
               where no refund is requested
    technical  the product is not working as documented
    other      none of the above, or the message is unclear

  If two apply, choose the earlier one. If you cannot tell from
  the message alone, use "other" rather than guessing.

  Reply with one JSON object and nothing else:
    {"queue": "refund|billing|technical|other", "confident": true|false}

what changed                          why
dropped "expert with 20 years"        no measurable effect, costs tokens
dropped "this is very important"      same
dropped "think step by step"          untested reflex; measure before adding it
task stated in one line, first        the model knows what it is doing by line one
message wrapped in tags               a message containing "ignore the above"
                                      is now data, not an instruction
labels given definitions              "billing, where no refund is requested"
                                      is the boundary the errors were sitting on
tie-break rule made explicit          removes the largest source of disagreement
                                      between two runs of the same prompt
"other" defined as the escape         without it the model guesses on unclear input
output contract at the end            exact keys, closed value list, nothing else
confident flag added                  gives you a cheap route to a human

Change one of those at a time and score after each, or you will finish the afternoon knowing the prompt got better and not knowing which line did it. The tie-break rule and the boundary definition usually account for most of the gain on a task like this, because they are the two places where a human labeller would also have hesitated.

Things that have aged badly#

• Offering the model a tip or threatening it. Measured effects were always marginal and are now noise.
• Long preambles about how important the task is. Tokens spent on nothing.
• Stacking superlatives about expertise. Specificity helps; flattery does not.
• Elaborate jailbreak-style framings to get ordinary work done. Usually a sign the task was underspecified.
• Asking for a confidence score and treating it as a probability. It is a token the model produced, not a calibrated estimate, and it is useful only as a rough flag for review.
• Telling the model not to hallucinate. Naming the failure does not prevent it; giving the model a way to say it does not know sometimes does.