AI system design interview questions
7 questions across 3 sections, in the order you would learn them. Every answer here is one that works — but an answer you have read is not an answer you can give, so use Test yourself and say it out loud first.
A structure for the design round
Read the section that teaches thisIf this comes up in an interview
Design a system that answers questions over a company's internal documents.
Clarify first: corpus size, freshness, latency, whether citations are required, budget. Then ingestion and structural chunking, hybrid retrieval with reranking, generation with citations and a threshold for declining when retrieval is weak. Then failure handling, per-request cost with the levers, and a retrieval eval separate from a generation eval. Finish with what I would build first.
Then they push further
Different employees can see different documents. How does that change the design?
Permissions have to be enforced at retrieval. Store access data with each chunk and filter by the user's permissions inside the search query, never by asking the model to ignore documents. Keep the index in sync when permissions change.
How do you know it is working after launch?
Log every trace, read a sample weekly, track feedback and whether people open the cited sources, and run the eval set on every change. A rising 'I do not know' rate often means the index has gone stale.
Answers take 8 seconds and people are complaining. Where do you look?
Break the time down by stage: retrieval, reranking, time to first token and generation. Then stream the answer, cut the context, cache frequent queries, and see whether reranking or a query rewriting call can be dropped or run in parallel.
What gets marked up. Strong answers ask about scale and requirements first, then cover ingestion, permissions, retrieval, cited generation, evaluation and monitoring. Weak answers stop at embed, store, retrieve, generate.
How do you handle the model being wrong in a production system?
Assume it will be. Show sources so users can verify, make correction cheap, keep a human in the loop where being wrong is expensive, gate destructive actions behind confirmation, and log everything so a bad answer can be traced. Design for wrong answers rather than trying to eliminate them.
Then they push further
Give a concrete example of that design.
For a refund assistant: the model drafts a decision with the policy it relied on, code checks the amount against the order and the policy limit, and anything over a threshold goes to a person. Users see the policy cited, so they can check it themselves.
What gets marked up. Marked up for designing on the assumption the model will sometimes be wrong: checks in code, human review where the cost is high, and errors that are visible and recoverable.
How would you know the system had regressed?
A fixed eval set run on every change, reporting quality, cost and latency together, with a held-out split. Plus production monitoring on the same signals, and sampling real traffic back into the eval set so it keeps describing what the system is actually asked.
Worked design: questions over company documents
Read the section that teaches thisIf this comes up in an interview
How do you handle document permissions in a RAG system?
Store permission groups as chunk metadata and filter at retrieval time, so the model never receives text the user may not see. Instructing the model to withhold restricted content is not a control, since anything in the context can leak.
What would you build first?
Ingestion, retrieval with permission filtering, and a retrieval eval, because retrieval quality and access control decide whether the system works. Generation is comparatively easy once the right chunks arrive.
Worked design: an agent acting on customer accounts
Read the section that teaches thisIf this comes up in an interview
How do you stop a support agent issuing unauthorised refunds?
Architecturally: a narrow refund tool with a threshold above which it returns a proposal requiring human approval, idempotency keys so retries cannot double-refund, and never letting customer text authorise anything. Plus an audit log of every action and approver.
How would you evaluate this agent?
Against the end state of test accounts rather than the agent's words: whether the correct action happened, for the right amount, exactly once. Include adversarial cases where the customer tries to manipulate the agent into an unauthorised action.
These are the questions for one topic. The whole question bank covers every topic on the site.