Testing a system with a non-deterministic part

Teams look at an AI system, note that the model is unpredictable, and conclude that the whole thing cannot be tested. That is how a codebase ends up with no tests at all around a great deal of ordinary, deterministic, buggy code. This section is about drawing the line in the right place: fake the model at the boundary and test everything either side of it — parsing, retrieval, tool execution, and the prompt assembly nobody checks and which is frequently not what you intended. It also marks where tests stop and evals start, because they are different jobs.

The judgement: what earns a test#

Not every line deserves a test, and a suite that tries produces the thing everyone hates: hundreds of tests that break on every refactor and catch nothing. The question to ask of any piece of code is whether it can be wrong in a way you would not notice. A function with branches, arithmetic, a boundary or a parsing step can. A function that reads an attribute cannot.

Code                                        Test it?
Chunking a document at a token boundary     yes: off-by-one, empty input, one huge word
Retry and backoff logic                     yes: does it stop, does it retry the right errors
Cost and token arithmetic                   yes: silently wrong forever otherwise
Parsing a model's JSON reply                yes: this is where production breaks
Prompt assembly after retrieval             yes: wrong and silent, see below
A dataclass with no methods                 no
A wrapper that forwards to the SDK          no: you would be testing the SDK
Whether the model's answer is good          no: that is an eval, not a test
Whether the model returns valid JSON        no: that is the provider's behaviour, not yours
That your logging calls logger.info         no: nothing breaks if it does not

The two rows at the bottom are the ones people get wrong in both directions. Do not test that the model returns valid JSON; that is not your code and you cannot fix it. Do test that your parser survives when it does not, because that is your code and you can.

The second question, once something is worth testing, is what to assert. Assert on behaviour the caller depends on, not on how it was produced. A test that asserts a private helper was called twice fails the moment somebody improves the helper, which teaches everyone that failing tests are normal, which is how a suite dies.

# Coupled to the implementation. Breaks on every refactor, catches nothing.
def test_chunking():
    with patch("app.chunk._split_on_sentences") as split:
        chunk(text)
        assert split.call_count == 3

# Coupled to the contract. Survives a rewrite, catches a real bug.
def test_chunks_never_exceed_the_limit():
    chunks = chunk(long_text, max_tokens=200)
    assert all(count_tokens(c) <= 200 for c in chunks)
    assert "".join(chunks) == long_text          # nothing lost, nothing duplicated

Most of an AI system is ordinary software#

Teams often decide their system cannot be tested because the model is unpredictable, and skip testing entirely. That is a mistake: the model is one component, and everything around it is deterministic and full of bugs.

• Parsing and validating model responses, including the malformed cases.
• Retry logic: does it back off, does it stop, does it retry the right errors.
• Budget enforcement: turns, cost, time.
• Tool dispatch: right function, right arguments, failures handled.
• Context assembly: what actually ends up in the prompt.
• Cost calculation.

Fake the model, test everything else#

class FakeModel:
    """Returns scripted replies so the logic around it is testable."""
    def __init__(self, replies):
        self.replies = list(replies)
        self.calls = []

    def chat(self, messages, **kwargs):
        self.calls.append(messages)
        return self.replies.pop(0)

def test_stops_at_turn_limit():
    model = FakeModel([tool_call("search"), tool_call("search"), tool_call("search")])
    result = Agent(model, max_turns=2).run("anything")

    assert result.stopped_because == "turn limit"
    assert len(model.calls) == 2

def test_retries_then_succeeds():
    model = FakeModel([RateLimited(), RateLimited(), reply("done")])
    assert Agent(model).run("x").text == "done"

Fixtures, and why test data is where suites rot#

A fixture is a named piece of setup that pytest builds and hands to any test that asks for it by parameter name. They are the best feature of the framework and the most common cause of a suite nobody trusts, and the reason is always the same: a fixture that is built once and shared, holding something a test can mutate.

import pytest

@pytest.fixture(scope="module")        # built once for the whole file
def docs():
    return [{"id": "a", "text": "alpha", "tags": []}]

def test_tagging_adds_a_tag(docs):
    docs[0]["tags"].append("seen")     # mutates the shared object
    assert docs[0]["tags"] == ["seen"]

def test_docs_start_untagged(docs):
    assert docs[0]["tags"] == []

$ pytest test_fixture_rot.py -q
.F
E       AssertionError: assert ['seen'] == []
1 failed, 1 passed

$ pytest test_fixture_rot.py::test_docs_start_untagged -q
1 passed

The fix is two rules. Default to function scope, which is what pytest gives you if you say nothing, so every test gets a fresh object. And when a fixture is expensive enough that you really do want to build it once, make it a factory: the fixture returns a function, and each test calls it to build its own.

import pytest

@pytest.fixture
def make_doc():
    """A factory, not a value. Each test builds its own, so nothing leaks."""
    def _make(doc_id="a", text="alpha", tags=None):
        return {"id": doc_id, "text": text, "tags": list(tags or [])}
    return _make

def test_docs_start_untagged(make_doc):
    assert make_doc()["tags"] == []

def test_a_document_with_awkward_text(make_doc):
    doc = make_doc(text="")                    # override only what matters here
    assert doc["text"] == ""

The list of tags being a new list on every call is deliberate, and it is the same bug as the mutable default argument in Python. Writing tags=[] in the signature shares one list between every document the factory ever makes, and you are back where you started.

Two more habits keep test data from turning into a second codebase. Build objects in the test that needs them rather than in a shared module of sample data, which nobody can change once forty tests depend on its exact contents. And keep fixture files small and hand-written: a fifty-kilobyte JSON blob captured from production is unreadable, so nobody can tell which part of it the test depends on, and it quietly becomes untouchable.

Testing a pipeline that touches a database#

The instinct is to mock the database, and it is usually wrong. Mocking it means your test asserts that you called a method, not that the data ended up correct, so it passes with a broken query, a missing constraint or a transaction that never commits. Those are exactly the bugs a database test exists to catch. Use a real database and make it disposable.

The pattern is one fixture that creates a schema, yields a connection, and throws the whole thing away afterwards. In-memory SQLite is free and fast enough to run on every save; a Postgres container from your compose file is the honest version when you use anything Postgres-specific, and the test code is identical.

import sqlite3
import pytest

SCHEMA = """
CREATE TABLE chunk (
    id INTEGER PRIMARY KEY,
    doc_id TEXT NOT NULL,
    text TEXT NOT NULL,
    UNIQUE (doc_id, text)
);
"""

def ingest(conn, doc_id, chunks):
    """Insert chunks, skipping any already stored for this document."""
    cur = conn.executemany(
        "INSERT OR IGNORE INTO chunk (doc_id, text) VALUES (?, ?)",
        [(doc_id, c) for c in chunks],
    )
    return cur.rowcount

@pytest.fixture
def conn():
    """A real database, schema and all, thrown away after each test."""
    c = sqlite3.connect(":memory:")
    c.executescript(SCHEMA)
    yield c
    c.close()

def test_ingest_stores_every_chunk(conn):
    ingest(conn, "doc-1", ["alpha", "beta"])
    assert conn.execute("SELECT count(*) FROM chunk").fetchone()[0] == 2

def test_ingest_is_idempotent(conn):
    ingest(conn, "doc-1", ["alpha", "beta"])
    ingest(conn, "doc-1", ["alpha", "beta"])       # the reprocessing case
    assert conn.execute("SELECT count(*) FROM chunk").fetchone()[0] == 2

def test_the_same_text_in_two_documents_is_kept_twice(conn):
    ingest(conn, "doc-1", ["alpha"])
    ingest(conn, "doc-2", ["alpha"])
    assert conn.execute("SELECT count(*) FROM chunk").fetchone()[0] == 2

Idempotency is the assertion to reach for on any pipeline, because the second run is not hypothetical. Ingestion jobs get retried, a queue redelivers, somebody reruns yesterday. Whatever the pipeline does, doing it twice should leave the same rows, and that behaviour lives in a UNIQUE constraint and an INSERT that respects it, not in a comment saying do not run this twice.

For a real database, the faster pattern is one schema per test session and a transaction per test that is rolled back at the end, so nothing is ever committed and no table has to be recreated. Whichever you choose, the rule is that no test may depend on a row another test left behind. If your suite fails when run in a different order, that is what it is telling you.

Testing something whose output is probabilistic#

This is the interview question, and the answer that scores is that you do not assert the output. You assert the things that must hold for every output the model could possibly produce, and you measure the rest.

Invariants first. These are ordinary unit tests and they must never flake: the label is one of the labels you allow, the JSON parses or degrades to a named fallback, the citation points at a document that exists, the answer is under the length limit, nothing in the output is a customer's email address. Run them against many different model outputs, including deliberately awful ones.

import json, random

LABELS = {"billing", "technical", "other"}

class WobblyModel:
    """Stands in for a real model: mostly right, occasionally not."""
    def __init__(self, seed=0):
        self.rng = random.Random(seed)

    def chat(self, messages):
        if self.rng.random() < 0.2:
            return "Sure! Here is the JSON: {'label': 'other'}"   # wrong quoting
        return json.dumps({"label": self.rng.choice(sorted(LABELS))})

def classify(model, text):
    """The deterministic wrapper. This is the part under test."""
    raw = model.chat([{"role": "user", "content": text}])
    start, end = raw.find("{"), raw.rfind("}")
    if start == -1 or end == -1:
        return {"label": "other", "parsed": False}
    try:
        data = json.loads(raw[start : end + 1])
    except json.JSONDecodeError:
        return {"label": "other", "parsed": False}
    label = data.get("label")
    return {"label": label if label in LABELS else "other", "parsed": True}

def test_label_is_always_one_of_the_allowed_labels():
    for seed in range(200):                      # 200 different model outputs
        assert classify(WobblyModel(seed), "my card was declined")["label"] in LABELS

def test_unparseable_output_degrades_instead_of_raising():
    class Garbage:
        def chat(self, messages):
            return "I cannot help with that."
    assert classify(Garbage(), "x") == {"label": "other", "parsed": False}

def test_parse_failure_is_visible_rather_than_silent():
    failures = sum(not classify(WobblyModel(s), "x")["parsed"] for s in range(500))
    assert 0 < failures < 500                    # allowed to fail, not to fail quietly

That parsed flag is doing more work than it looks. A wrapper that quietly returns the fallback label is indistinguishable, from the outside, from one that is working, so a provider change that breaks your parsing shows up as a slow drift in quality nobody can explain. Return the failure as data, count it, and alert on the rate.

Then, and only then, a scored run over a fixed set with a threshold. This is an eval, not a unit test: it costs money, it runs on a schedule rather than on every commit, and it reports a number. The thing that surprises people is how easy it is to make that number flake, and the arithmetic is worth doing once.

import numpy as np

rng = np.random.default_rng(0)
TRUE_RATE = 0.90        # the system really does pass 90 percent of the time

for n in (20, 200, 1000):
    scores = rng.binomial(n, TRUE_RATE, size=200_000) / n
    for threshold in (0.90, 0.85, 0.80):
        fails = (scores < threshold).mean()
        print(f"n={n:5d}  threshold={threshold:.2f}  suite fails {fails:6.1%} of runs")
    print()

n=   20  threshold=0.90  suite fails  32.2% of runs
n=   20  threshold=0.85  suite fails  13.2% of runs
n=   20  threshold=0.80  suite fails   4.3% of runs

n=  200  threshold=0.90  suite fails  44.1% of runs
n=  200  threshold=0.85  suite fails   1.0% of runs
n=  200  threshold=0.80  suite fails   0.0% of runs

n= 1000  threshold=0.90  suite fails  47.3% of runs
n= 1000  threshold=0.85  suite fails   0.0% of runs
n= 1000  threshold=0.80  suite fails   0.0% of runs

Read the first column of each block. A threshold set at the rate the system genuinely achieves fails about half the time, and adding examples makes it worse, not better, because the score converges on the true rate from both sides. More examples do not rescue a threshold set at the true rate; they let you set the threshold closer to it. At twenty examples you need to sit five points below to be safe. At two hundred, one point of margin costs you a one percent false alarm rate.

So the rule for a threshold is: measure the current rate, set the gate a few points below it, and fix the task set so today's number and next week's are comparable. And say the word regression rather than quality. The gate is there to catch the change that made things worse, not to certify that the system is good.

Where tests end#

A test asserts something that must always be true. Whether the model's answer is good is not that kind of claim, so it belongs in an eval, which measures a rate rather than asserting a fact. Keeping the two separate is what makes each of them trustworthy.

Unit test   deterministic, must always pass, runs in milliseconds
Eval        graded, reports a rate, costs money, runs on a schedule

A flaky unit test gets deleted. An eval that scores 0.86 is fine.

The practical consequence is that they belong in different places. Unit tests run on every commit and block the merge. Evals run nightly or before a release, on their own schedule, and report to a dashboard. Putting an eval in the unit test suite is how a team learns to rerun CI until it goes green, which costs them every real failure after that.

What to skip#

• A coverage target. It measures which lines ran, not whether anything was checked, and chasing ninety percent produces tests written to touch lines.
• Testing your wrappers around a provider SDK. You would be testing their code with your mock of it.
• Mocking the database. Use a real one; that is what the test is for.
• Mutation testing, contract testing and the rest, until you have a suite that people trust and run.
• Asserting on exact model output anywhere, ever, including in a snapshot. That is the one habit that guarantees the suite gets ignored.