Tracing & observability interview questions
5 questions across 2 sections, in the order you would learn them. Every answer here is one that works — but an answer you have read is not an answer you can give, so use Test yourself and say it out loud first.
Seeing what the system did
Read the section that teaches thisIf this comes up in an interview
How do you debug an agent that behaved oddly in production?
Pull the trace for that run. Read what was actually sent at each turn, which tools were called with which arguments, what came back, and where it diverged. Most surprising behaviour is obvious once you see the rendered prompt rather than the template.
What would you log on every model call?
A run id to tie the run together, the step and turn, model, input and output tokens, cached tokens, cost, latency and stop reason. For tools, the name, arguments, success, error and result size.
Why record cached token counts?
It is the only way to tell whether prompt caching is working. If cached tokens are consistently zero, something volatile has crept into the prompt prefix and your largest cost saving is silently off.
Using traces to improve the system
Read the section that teaches thisIf this comes up in an interview
How do you use production traces to improve an AI system?
Keep failures, slow and expensive runs, and a sample of normal ones. Categorise failures and fix the largest category first, and add each real failure to the eval set so it cannot silently return.
How useful is thumbs-up and thumbs-down feedback?
Useful as a pointer to where to look, and noisy and biased as a measure, since only some users click. Behavioural signals such as retries and abandonment are often more informative.
These are the questions for one topic. The whole question bank covers every topic on the site.