Testing

A test is a small piece of code that runs your code and checks the answer. The reason to write them is not diligence: it is that they let you change something six months later and find out in seconds whether you broke anything. This section covers pytest, running the same logic against many inputs, sharing setup between tests, and standing in fakes for slow or paid dependencies — which matters in AI work, where a real model call costs money and answers differently each time.

Your first test#

A test is a function whose name starts with test_ containing a plain assert. pytest finds them and runs them. There is no base class and no special assertion methods.

# add.py
def add(a, b):
    return a + b

# test_add.py
from add import add

def test_adds_two_numbers():
    assert add(2, 3) == 5

def test_handles_negatives():
    assert add(-1, 1) == 0

pytest                  # run everything
pytest test_add.py      # one file
pytest -k "negative"    # only tests matching a name
pytest -v               # show each test name

Many inputs, one test#

parametrize runs the same test once per set of inputs, and reports each one separately when it fails. This replaces copying a test five times with different values.

import pytest

@pytest.mark.parametrize(
    "raw, expected",
    [
        ("42", 42),
        (" 42 ", 42),
        ("0", 0),
    ],
)
def test_parses_int(raw, expected):
    assert parse_int(raw) == expected

Testing that something raises#

def test_rejects_bad_temperature():
    with pytest.raises(ValueError):
        set_temperature(5)

def test_error_message_is_useful():
    with pytest.raises(ValueError, match="must be 0 to 2"):
        set_temperature(5)

Fixtures#

A fixture is setup shared between tests. Declare it once, then take it as an argument in any test that needs it.

@pytest.fixture
def sample_records():
    return [
        {"id": "1", "score": 0.9},
        {"id": "2", "score": 0.4},
    ]

def test_filters_by_score(sample_records):
    result = above(sample_records, 0.5)
    assert len(result) == 1

Fakes, not mocks everywhere#

Replace the thing at the edge of your system, such as the model client, with a simple object that returns fixed answers. Do not replace your own functions with mocks. A test that only checks a mock was called proves nothing and keeps passing after you break the behaviour.

class FakeModel:
    def __init__(self, replies):
        self.replies = list(replies)
        self.calls = 0

    def call(self, prompt):
        self.calls += 1
        return self.replies.pop(0)

def test_agent_stops_at_turn_limit():
    model = FakeModel(["think", "think", "think"])
    agent = Agent(model=model, max_turns=2)

    agent.run("anything")

    assert model.calls == 2

What you can test in an AI system#

More than people expect. The model's output is not predictable, but everything around it is, and that is where most bugs live.

• Testable normally: parsing responses, retry logic, budget and turn limits, tool dispatch, building the message list, cost calculation.
• Not unit testable: whether the model's answer is good. That needs evals, which are a different tool for a different job.