Asked about A structure for the design round
Design a system that answers questions over a company's internal documents.
An answer that works
Clarify first: corpus size, freshness, latency, whether citations are required, budget. Then ingestion and structural chunking, hybrid retrieval with reranking, generation with citations and a threshold for declining when retrieval is weak. Then failure handling, per-request cost with the levers, and a retrieval eval separate from a generation eval. Finish with what I would build first.
Then they push further
Different employees can see different documents. How does that change the design?
Permissions have to be enforced at retrieval. Store access data with each chunk and filter by the user's permissions inside the search query, never by asking the model to ignore documents. Keep the index in sync when permissions change.
How do you know it is working after launch?
Log every trace, read a sample weekly, track feedback and whether people open the cited sources, and run the eval set on every change. A rising 'I do not know' rate often means the index has gone stale.
Answers take 8 seconds and people are complaining. Where do you look?
Break the time down by stage: retrieval, reranking, time to first token and generation. Then stream the answer, cut the context, cache frequent queries, and see whether reranking or a query rewriting call can be dropped or run in parallel.
What gets marked up
Strong answers ask about scale and requirements first, then cover ingestion, permissions, retrieval, cited generation, evaluation and monitoring. Weak answers stop at embed, store, retrieve, generate.
Reading an answer is not being able to give one. The section that teaches this is A structure for the design round, 5 hours of AI system design.
Learn this properlyAlso asked about AI system design