Testing interview questions
5 questions across 2 sections, in the order you would learn them. Every answer here is one that works — but an answer you have read is not an answer you can give, so use Test yourself and say it out loud first.
Testing a system with a non-deterministic part
Read the section that teaches thisIf this comes up in an interview
How do you test code that calls an LLM?
Fake the model at the boundary with scripted replies, which makes everything around it deterministic: parsing, retries, budgets, tool dispatch, prompt assembly. The quality of the model's output is not a unit test; that is what evals are for.
What is usually untested in an AI system and should not be?
Prompt assembly. What actually ends up in the prompt after retrieval, history management and compaction is often not what was intended, and the failure is silent. Asserting that the task survives compaction and that the cacheable prefix is stable catches real bugs.
What is the difference between a test and an eval?
A test asserts something that must always hold and is deterministic. An eval measures a rate over a task set and is expected to be imperfect. Mixing them produces flaky tests that get deleted and evals nobody trusts.
Tests for prompts, tools and agent loops
Read the section that teaches thisIf this comes up in an interview
How do you test an agent without calling a model?
Replace the model with a scripted fake that returns a planned sequence of tool calls and answers. Then test the loop's behaviour: tools run with the right arguments, results are fed back, budgets and repeat guards trigger, errors become tool results. Model quality is left to evals.
Then they push further
Is that not testing a fantasy? The real model will not follow your script.
It tests your code, which is deterministic and should be correct for any sequence the model might produce. Scripts that include odd sequences, such as repeated calls or unknown tools, are exactly what the real model will eventually do.
What gets marked up. Marked up for a clear split: tests for the deterministic loop, evals for model behaviour.
What is property-based testing and where does it help in AI code?
Generating many random inputs and checking a property holds for all of them. It suits parsers and validators of model output, which must never crash on strange input.
These are the questions for one topic. The whole question bank covers every topic on the site.