Testing

Teams look at an AI system, note that the model is unpredictable, and conclude that the whole thing cannot be tested. That is how a codebase ends up with no tests at all around a great deal of ordinary, deterministic, buggy code. This section is about drawing the line in the right place: fake the model at the boundary and test everything either side of it — parsing, retrieval, tool execution, and the prompt assembly nobody checks and which is frequently not what you intended. It also marks where tests stop and evals start, because they are different jobs.

The judgement: what earns a test#

Not every line deserves a test, and a suite that tries produces the thing everyone hates: hundreds of tests that break on every refactor and catch nothing. The question to ask of any piece of code is whether it can be wrong in a way you would not notice. A function with branches, arithmetic, a boundary or a parsing step can. A function that reads an attribute cannot.

Code                                        Test it?
Chunking a document at a token boundary     yes: off-by-one, empty input, one huge word
Retry and backoff logic                     yes: does it stop, does it retry the right errors
Cost and token arithmetic                   yes: silently wrong forever otherwise
Parsing a model's JSON reply                yes: this is where production breaks
Prompt assembly after retrieval             yes: wrong and silent, see below
A dataclass with no methods                 no
A wrapper that forwards to the SDK          no: you would be testing the SDK
Whether the model's answer is good          no: that is an eval, not a test
Whether the model returns valid JSON        no: that is the provider's behaviour, not yours
That your logging calls logger.info         no: nothing breaks if it does not

The two rows at the bottom are the ones people get wrong in both directions. Do not test that the model returns valid JSON; that is not your code and you cannot fix it. Do test that your parser survives when it does not, because that is your code and you can.

The second question, once something is worth testing, is what to assert. Assert on behaviour the caller depends on, not on how it was produced. A test that asserts a private helper was called twice fails the moment somebody improves the helper, which teaches everyone that failing tests are normal, which is how a suite dies.

# Coupled to the implementation. Breaks on every refactor, catches nothing.
def test_chunking():
    with patch("app.chunk._split_on_sentences") as split:
        chunk(text)
        assert split.call_count == 3

# Coupled to the contract. Survives a rewrite, catches a real bug.
def test_chunks_never_exceed_the_limit():
    chunks = chunk(long_text, max_tokens=200)
    assert all(count_tokens(c) <= 200 for c in chunks)
    assert "".join(chunks) == long_text          # nothing lost, nothing duplicated

Most of an AI system is ordinary software#

Teams often decide their system cannot be tested because the model is unpredictable, and skip testing entirely. That is a mistake: the model is one component, and everything around it is deterministic and full of bugs.

• Parsing and validating model responses, including the malformed cases.
• Retry logic: does it back off, does it stop, does it retry the right errors.
• Budget enforcement: turns, cost, time.
• Tool dispatch: right function, right arguments, failures handled.
• Context assembly: what actually ends up in the prompt.
• Cost calculation.

Fake the model, test everything else#

class FakeModel:
    """Returns scripted replies so the logic around it is testable."""
    def __init__(self, replies):
        self.replies = list(replies)
        self.calls = []

    def chat(self, messages, **kwargs):
        self.calls.append(messages)
        return self.replies.pop(0)

def test_stops_at_turn_limit():
    model = FakeModel([tool_call("search"), tool_call("search"), tool_call("search")])
    result = Agent(model, max_turns=2).run("anything")

    assert result.stopped_because == "turn limit"
    assert len(model.calls) == 2

def test_retries_then_succeeds():
    model = FakeModel([RateLimited(), RateLimited(), reply("done")])
    assert Agent(model).run("x").text == "done"

Fixtures, and why test data is where suites rot#

A fixture is a named piece of setup that pytest builds and hands to any test that asks for it by parameter name. They are the best feature of the framework and the most common cause of a suite nobody trusts, and the reason is always the same: a fixture that is built once and shared, holding something a test can mutate.

import pytest

@pytest.fixture(scope="module")        # built once for the whole file
def docs():
    return [{"id": "a", "text": "alpha", "tags": []}]

def test_tagging_adds_a_tag(docs):
    docs[0]["tags"].append("seen")     # mutates the shared object
    assert docs[0]["tags"] == ["seen"]

def test_docs_start_untagged(docs):
    assert docs[0]["tags"] == []

$ pytest test_fixture_rot.py -q
.F
E       AssertionError: assert ['seen'] == []
1 failed, 1 passed

$ pytest test_fixture_rot.py::test_docs_start_untagged -q
1 passed

The fix is two rules. Default to function scope, which is what pytest gives you if you say nothing, so every test gets a fresh object. And when a fixture is expensive enough that you really do want to build it once, make it a factory: the fixture returns a function, and each test calls it to build its own.

import pytest

@pytest.fixture
def make_doc():
    """A factory, not a value. Each test builds its own, so nothing leaks."""
    def _make(doc_id="a", text="alpha", tags=None):
        return {"id": doc_id, "text": text, "tags": list(tags or [])}
    return _make

def test_docs_start_untagged(make_doc):
    assert make_doc()["tags"] == []

def test_a_document_with_awkward_text(make_doc):
    doc = make_doc(text="")                    # override only what matters here
    assert doc["text"] == ""

The list of tags being a new list on every call is deliberate, and it is the same bug as the mutable default argument in Python. Writing tags=[] in the signature shares one list between every document the factory ever makes, and you are back where you started.

Two more habits keep test data from turning into a second codebase. Build objects in the test that needs them rather than in a shared module of sample data, which nobody can change once forty tests depend on its exact contents. And keep fixture files small and hand-written: a fifty-kilobyte JSON blob captured from production is unreadable, so nobody can tell which part of it the test depends on, and it quietly becomes untouchable.

Testing a pipeline that touches a database#

The instinct is to mock the database, and it is usually wrong. Mocking it means your test asserts that you called a method, not that the data ended up correct, so it passes with a broken query, a missing constraint or a transaction that never commits. Those are exactly the bugs a database test exists to catch. Use a real database and make it disposable.

The pattern is one fixture that creates a schema, yields a connection, and throws the whole thing away afterwards. In-memory SQLite is free and fast enough to run on every save; a Postgres container from your compose file is the honest version when you use anything Postgres-specific, and the test code is identical.

import sqlite3
import pytest

SCHEMA = """
CREATE TABLE chunk (
    id INTEGER PRIMARY KEY,
    doc_id TEXT NOT NULL,
    text TEXT NOT NULL,
    UNIQUE (doc_id, text)
);
"""

def ingest(conn, doc_id, chunks):
    """Insert chunks, skipping any already stored for this document."""
    cur = conn.executemany(
        "INSERT OR IGNORE INTO chunk (doc_id, text) VALUES (?, ?)",
        [(doc_id, c) for c in chunks],
    )
    return cur.rowcount

@pytest.fixture
def conn():
    """A real database, schema and all, thrown away after each test."""
    c = sqlite3.connect(":memory:")
    c.executescript(SCHEMA)
    yield c
    c.close()

def test_ingest_stores_every_chunk(conn):
    ingest(conn, "doc-1", ["alpha", "beta"])
    assert conn.execute("SELECT count(*) FROM chunk").fetchone()[0] == 2

def test_ingest_is_idempotent(conn):
    ingest(conn, "doc-1", ["alpha", "beta"])
    ingest(conn, "doc-1", ["alpha", "beta"])       # the reprocessing case
    assert conn.execute("SELECT count(*) FROM chunk").fetchone()[0] == 2

def test_the_same_text_in_two_documents_is_kept_twice(conn):
    ingest(conn, "doc-1", ["alpha"])
    ingest(conn, "doc-2", ["alpha"])
    assert conn.execute("SELECT count(*) FROM chunk").fetchone()[0] == 2

Idempotency is the assertion to reach for on any pipeline, because the second run is not hypothetical. Ingestion jobs get retried, a queue redelivers, somebody reruns yesterday. Whatever the pipeline does, doing it twice should leave the same rows, and that behaviour lives in a UNIQUE constraint and an INSERT that respects it, not in a comment saying do not run this twice.

For a real database, the faster pattern is one schema per test session and a transaction per test that is rolled back at the end, so nothing is ever committed and no table has to be recreated. Whichever you choose, the rule is that no test may depend on a row another test left behind. If your suite fails when run in a different order, that is what it is telling you.

Testing something whose output is probabilistic#

This is the interview question, and the answer that scores is that you do not assert the output. You assert the things that must hold for every output the model could possibly produce, and you measure the rest.

Invariants first. These are ordinary unit tests and they must never flake: the label is one of the labels you allow, the JSON parses or degrades to a named fallback, the citation points at a document that exists, the answer is under the length limit, nothing in the output is a customer's email address. Run them against many different model outputs, including deliberately awful ones.

import json, random

LABELS = {"billing", "technical", "other"}

class WobblyModel:
    """Stands in for a real model: mostly right, occasionally not."""
    def __init__(self, seed=0):
        self.rng = random.Random(seed)

    def chat(self, messages):
        if self.rng.random() < 0.2:
            return "Sure! Here is the JSON: {'label': 'other'}"   # wrong quoting
        return json.dumps({"label": self.rng.choice(sorted(LABELS))})

def classify(model, text):
    """The deterministic wrapper. This is the part under test."""
    raw = model.chat([{"role": "user", "content": text}])
    start, end = raw.find("{"), raw.rfind("}")
    if start == -1 or end == -1:
        return {"label": "other", "parsed": False}
    try:
        data = json.loads(raw[start : end + 1])
    except json.JSONDecodeError:
        return {"label": "other", "parsed": False}
    label = data.get("label")
    return {"label": label if label in LABELS else "other", "parsed": True}

def test_label_is_always_one_of_the_allowed_labels():
    for seed in range(200):                      # 200 different model outputs
        assert classify(WobblyModel(seed), "my card was declined")["label"] in LABELS

def test_unparseable_output_degrades_instead_of_raising():
    class Garbage:
        def chat(self, messages):
            return "I cannot help with that."
    assert classify(Garbage(), "x") == {"label": "other", "parsed": False}

def test_parse_failure_is_visible_rather_than_silent():
    failures = sum(not classify(WobblyModel(s), "x")["parsed"] for s in range(500))
    assert 0 < failures < 500                    # allowed to fail, not to fail quietly

That parsed flag is doing more work than it looks. A wrapper that quietly returns the fallback label is indistinguishable, from the outside, from one that is working, so a provider change that breaks your parsing shows up as a slow drift in quality nobody can explain. Return the failure as data, count it, and alert on the rate.

Then, and only then, a scored run over a fixed set with a threshold. This is an eval, not a unit test: it costs money, it runs on a schedule rather than on every commit, and it reports a number. The thing that surprises people is how easy it is to make that number flake, and the arithmetic is worth doing once.

import numpy as np

rng = np.random.default_rng(0)
TRUE_RATE = 0.90        # the system really does pass 90 percent of the time

for n in (20, 200, 1000):
    scores = rng.binomial(n, TRUE_RATE, size=200_000) / n
    for threshold in (0.90, 0.85, 0.80):
        fails = (scores < threshold).mean()
        print(f"n={n:5d}  threshold={threshold:.2f}  suite fails {fails:6.1%} of runs")
    print()

n=   20  threshold=0.90  suite fails  32.2% of runs
n=   20  threshold=0.85  suite fails  13.2% of runs
n=   20  threshold=0.80  suite fails   4.3% of runs

n=  200  threshold=0.90  suite fails  44.1% of runs
n=  200  threshold=0.85  suite fails   1.0% of runs
n=  200  threshold=0.80  suite fails   0.0% of runs

n= 1000  threshold=0.90  suite fails  47.3% of runs
n= 1000  threshold=0.85  suite fails   0.0% of runs
n= 1000  threshold=0.80  suite fails   0.0% of runs

Read the first column of each block. A threshold set at the rate the system genuinely achieves fails about half the time, and adding examples makes it worse, not better, because the score converges on the true rate from both sides. More examples do not rescue a threshold set at the true rate; they let you set the threshold closer to it. At twenty examples you need to sit five points below to be safe. At two hundred, one point of margin costs you a one percent false alarm rate.

So the rule for a threshold is: measure the current rate, set the gate a few points below it, and fix the task set so today's number and next week's are comparable. And say the word regression rather than quality. The gate is there to catch the change that made things worse, not to certify that the system is good.

Where tests end#

A test asserts something that must always be true. Whether the model's answer is good is not that kind of claim, so it belongs in an eval, which measures a rate rather than asserting a fact. Keeping the two separate is what makes each of them trustworthy.

Unit test   deterministic, must always pass, runs in milliseconds
Eval        graded, reports a rate, costs money, runs on a schedule

A flaky unit test gets deleted. An eval that scores 0.86 is fine.

The practical consequence is that they belong in different places. Unit tests run on every commit and block the merge. Evals run nightly or before a release, on their own schedule, and report to a dashboard. Putting an eval in the unit test suite is how a team learns to rerun CI until it goes green, which costs them every real failure after that.

What to skip#

• A coverage target. It measures which lines ran, not whether anything was checked, and chasing ninety percent produces tests written to touch lines.
• Testing your wrappers around a provider SDK. You would be testing their code with your mock of it.
• Mocking the database. Use a real one; that is what the test is for.
• Mutation testing, contract testing and the rest, until you have a suite that people trust and run.
• Asserting on exact model output anywhere, ever, including in a snapshot. That is the one habit that guarantees the suite gets ignored.

The model's output is not deterministic, which leads people to conclude that AI systems cannot be unit tested. Most of the system is deterministic: the loop that runs tools, the budget that stops it, the parser that reads its output, the code that assembles the prompt. This section tests all of that by replacing the model with a fake that returns a script you wrote, so you can drive the loop through failure, repetition and exhaustion without spending anything or waiting on a network.

A fake that follows a script#

The agent loop has real logic: run tools, stop on budget, handle errors, detect repeats. To test it, replace the model with a fake that returns a scripted sequence of responses.

class ScriptedModel:
    def __init__(self, responses):
        self.responses = list(responses)
        self.calls = []

    def generate(self, messages, tools):
        self.calls.append(messages)
        return self.responses.pop(0)

def test_agent_runs_tool_then_answers():
    model = ScriptedModel([
        ToolCall("lookup_order", {"id": "A-102"}),
        FinalAnswer("Your order shipped yesterday."),
    ])
    result = run_agent("Where is my order?", model=model, tools=FAKE_TOOLS)
    assert result.text == "Your order shipped yesterday."
    assert "A-102" in str(model.calls[1])     # the tool result was sent back

Note what the fake records as well as what it returns. Keeping the messages it was called with is what lets you assert on the thing nobody checks: that the tool result actually made it back into the conversation. A fake that only returns is half a fake.

Test the edges of the loop#

• The same tool call repeated three times triggers the repeat guard.
• A tool that raises produces an error result, not a crash.
• Hitting the turn budget returns a partial result with a reason.
• An unknown tool name is rejected before anything runs.

Mocking a model call properly#

Two mistakes make a mocked model call worthless, and both produce a green test. The first is patching the name where the class was defined rather than the name the code under test actually reads. Python binds names at import time, so a module that did from app.provider import Client holds its own reference, and patching app.provider.Client afterwards changes nothing it will ever look at.

# app/summarise.py
from app.provider import Client
client = Client()

def summarise(text):
    return client.chat([{"role": "user", "content": "Summarise: " + text}]).strip()


# the test
def test_patching_where_it_was_defined():
    with patch("app.provider.Client") as fake:          # wrong target
        fake.return_value.chat.return_value = "  short  "
        assert summarise("long text") == "short"

def test_patching_where_it_is_looked_up():
    with patch("app.summarise.client") as fake:         # right target
        fake.chat.return_value = "  short  "
        assert summarise("long text") == "short"

Prove to yourself that the first one does nothing by making the real client refuse to work. Give it a chat method that raises, and the wrong patch fails loudly instead of quietly reaching the network.

$ pytest test_patch_target.py -q
app/summarise.py:7: in summarise
    reply = client.chat([{"role": "user", "content": "Summarise: " + text}])
E       RuntimeError: real network call: this must never run in a test
1 failed, 1 passed

The second mistake is a plain MagicMock, which agrees with everything. It will accept a method that does not exist, arguments in the wrong order and keyword arguments you invented, and every assertion about it will pass. create_autospec builds the mock from the real class, so a typo or a changed signature fails the test instead of sailing through it.

from unittest.mock import MagicMock, create_autospec

def test_a_plain_mock_agrees_with_anything():
    fake = MagicMock()
    fake.chatt(messages=[], temperature="hot", nonsense=1)   # no such method
    assert fake.chatt.called            # passes, and proves nothing

def test_autospec_rejects_the_typo():
    fake = create_autospec(Client, instance=True)
    fake.chatt([])                      # AttributeError, as it should be

For the provider SDK specifically, prefer a small hand-written fake class over patching anything. It is the same amount of code, it documents the shape your system depends on in one readable place, and when the SDK changes your fake stops matching, which is information. A mock silently keeps agreeing with a signature that no longer exists.

Recorded responses#

For an integration test that should exercise the real API format, record one real response to a file and replay it in tests. Libraries such as vcrpy do this for HTTP. The test is fast, free and repeatable, and you rerecord deliberately when the API changes.

Two rules keep recordings from becoming a liability. Scrub the API key out of the recorded request headers before it is committed, because a recording is a verbatim copy of a real exchange. And put a date in the file and rerecord on a schedule, otherwise the suite is testing an API shape from eighteen months ago and passing confidently.

Let the computer invent inputs#

Property-based testing generates many random inputs and checks that a property always holds. It is excellent for parsers of model output, which meet strange input in production.

from hypothesis import given, strategies as st

@given(st.text())
def test_parser_never_crashes(raw):
    result = parse_model_output(raw)          # must return a result or a clean error
    assert result.ok or result.error

What makes it worth the import is shrinking. When it finds a failure it keeps simplifying the input until it has the smallest one that still breaks, so you are handed a minimal reproduction rather than a page of noise. Pointed at the parser everybody writes first, it reports the answer in one line.

def classify_naive(raw):
    """The version everyone writes first."""
    return json.loads(raw[raw.find("{") : raw.rfind("}") + 1])["label"]

@given(st.text())
def test_naive_parser_never_raises(raw):
    assert classify_naive(raw) in LABELS

Snapshot the prompt#

Prompt assembly code, which combines templates, retrieved text and history, breaks quietly. A snapshot test stores the assembled prompt for a fixed input and fails when it changes, so every change to the prompt is reviewed on purpose.

Snapshot the prompt you send, never the answer you get back. The prompt is your code and must be stable; the answer is the model's and never will be. And keep the snapshot small enough to read in a diff, because a snapshot nobody reads gets accepted with the update flag on reflex, which is the same as not having it.

What a flaky test is telling you#

A flaky test is not noise to be retried. It is a report that something in your system depends on an order, a clock or a machine that nobody declared, and most of the time the bug is in the code rather than the test. Here is one that is not obviously random at all.

def top_tags(docs):
    """Returns the distinct tags. The set is an implementation detail."""
    return list({t for d in docs for t in d["tags"]})

def test_top_tags():
    docs = [{"tags": ["rag", "eval", "prompt"]}]
    assert top_tags(docs) == ["rag", "eval", "prompt"]

$ for i in 1 2 3 4 5; do pytest test_flaky_set.py -q | tail -1; done
1 failed in 0.13s
1 passed in 0.13s
1 failed in 0.15s
1 failed in 0.14s
1 failed in 0.13s

E  AssertionError: assert ['eval', 'prompt', 'rag'] == ['rag', 'eval', 'prompt']

The instructive part is what it found. The test is wrong to assert an order, but the function is also wrong to return a list: it promises an order it does not have, and somewhere downstream a caller is showing the first three tags to a user and getting a different three every deploy. Fix both. Return a sorted list or a set, and assert on the same thing.

Flakes when                           What it is actually telling you
run in a different order              tests share state, or a fixture is mutable
run in parallel but not alone         shared file, port, database row or temp path
only in CI                            a clock, a locale, a timezone, or less memory
about once in twenty runs             a real race, or a set or dict order dependency
after the first of the month          a date fixture built from datetime.now
only on a colleague's machine         an undeclared dependency on the environment

What never to do is add a retry decorator to make it green. A retry hides the failure rate rather than removing it, and the underlying race is still there in production where nothing reruns. If a test genuinely cannot be made deterministic, delete it and write an eval instead, because that is what it was.

What to skip#

• Testing that the model is any good. Different job, different tool, different schedule.
• Mocking the HTTP layer by hand. Record it, or fake the client above it.
• Asserting on exact prompt text inside a unit test. Snapshot it once, in one place.
• Property-based tests on business logic. Keep them for parsers and validators, where the input space is genuinely unbounded.