Worked design: questions over company documents

A design round hands you a vague prompt and forty-five minutes to talk. This section is one such question worked through end to end: the clarifying questions and what their answers change, the architecture, what happens when each part fails, the cost arithmetic done out loud, and what you would build first. It is a walk-through to argue with rather than a template to memorise, and the value is in seeing the order the parts come in and how much of the design falls out of a single answer about permissions.

The prompt#

Design a system that lets employees ask questions about the company's internal documents. This is the most common AI design question. What follows is one reasonable walk-through, not the only correct one.

Clarify, five minutes#

How many documents, and how fast do they change?
    -> 200,000 documents, a few thousand updated each day
Who can see what?
    -> permissions vary by team; this is critical
How fresh must answers be?
    -> within a day is fine
Latency?
    -> a few seconds is acceptable, it is interactive
Citations needed?
    -> yes, people must be able to verify
Budget?
    -> assume cost matters but is not the only constraint

Size it before you draw it#

Two hundred thousand documents sounds large and is not. Work that out loud before touching the architecture, because the answer changes which problems you spend the next forty minutes on. Assume an average document of about 3,600 words, which is 4,800 tokens at the usual 1.33 tokens a word, and chunks of 400 tokens. That is twelve chunks a document and 2.4 million chunks in total.

Index size
  vectors   2.4M x 1,536 dims x 4 bytes  =  14.7 GB
  graph     2.4M x ~150 bytes (HNSW)     =   0.4 GB
  text      2.4M x 400 tok x ~4.5 bytes  =   4.3 GB
                                            ------
                                            ~19 GB

Query load
  10,000 queries a day over an 8-hour working day
      10,000 / 28,800  =  0.35 per second
      peak at 3x       =  ~1 per second

First build
  2.4M chunks x 400 tokens  =  960M tokens to embed
  at $0.02 per million      =  about $19, one off
  at 1M tokens a minute     =  16 hours

Steady state
  3,000 changed docs x 12 chunks x 400 tok = 14.4M tokens
                                           = about 30 cents a day

Having said it, say what it means: this is not a distributed systems problem. There is no sharding decision, no replication strategy and no queue worth discussing, and a candidate who spends ten minutes on those has misread the question. It is a quality problem, and every remaining minute should go to retrieval quality, permissions and evaluation. If the interviewer wanted the scaling conversation they would have said twenty million documents, and if they say it later, the answer is that the vectors no longer fit in memory, so you shard by tenant or quantise to int8 and take 14.7 GB down to 3.7.

Architecture#

1. 1.Ingestion: a daily job parses changed documents, keyed on content hash so unchanged ones are skipped.
2. 2.Chunking on structure, with document title and heading path attached to each chunk, and the permission groups stored as metadata.
3. 3.Index: hybrid, vector plus keyword, with metadata filtering so retrieval only ever returns chunks the user may see.
4. 4.Query: rewrite follow-up questions into standalone form, retrieve fifty, rerank to ten, send the top five with ids.
5. 5.Generation: answer only from the provided chunks, cite them, and decline when retrieval is weak.

Three of those five steps deserve a sentence of justification, because an interviewer will ask for it. Chunking on structure rather than on a fixed character count matters because a chunk that begins mid-sentence under a heading it cannot see is unretrievable by the question it answers; attaching the title and heading path is what makes a chunk about deployment findable by a question about deployment. Hybrid retrieval matters because vectors are bad at exact strings, and internal documents are full of them: error codes, ticket numbers, the name of a service. A pure vector index cannot reliably find the page for error 5031, and keyword search cannot find it when the user asks about the checkout failure, so you run both and fuse the rankings. Reranking matters because embedding similarity is cheap and approximate, and a cross-encoder reading the query and the chunk together is far more accurate but far too slow to run over the whole corpus, so you use the cheap one to get fifty candidates and the accurate one to pick five.

Query rewriting is the step candidates omit. The second question in a conversation is almost never standalone: what about last year retrieves nothing on its own. Rewriting it into a self-contained question using the conversation so far costs one small model call, and it is the difference between a demo and something people use twice.

Failure#

• No good match: threshold on retrieval score and say so rather than inventing an answer.
• Stale document: show the last-updated date with every citation.
• Model provider down: fall back to returning the top retrieved passages without generation.
• Prompt injection in a document: the generation step holds no tools and no permissions, so there is nothing for an injection to reach.

Cost, out loud#

Per query: ~2,500 input tokens (5 chunks + instructions)
           ~300 output tokens
At illustrative rates of $1 per million in, $5 per million out:
           ~$0.004 per query
10,000 queries a day ->  ~$40 a day, ~$1,200 a month

Levers: cache the stable instructions, trim chunk size,
route simple questions to a smaller model.

Then convert it into the unit the business cares about, because that is the sentence that ends the cost conversation. Four dollars per thousand queries, twelve hundred a month, five thousand employees: twenty-four cents per person per month. Nobody is going to cancel this over the model bill, and saying so tells the interviewer you know which costs are worth optimising. The reranker is the same story in reverse: it reads fifty chunks of 400 tokens, so twenty thousand tokens a query, which at cross-encoder prices is comparable to the generation call and buys more quality per penny than anything else on the list.

The number that would actually change the design is engineering time, not tokens. Say that. A design round rewards knowing which of your numbers matters.

Evaluation and what to build first#

Two evals: retrieval recall on questions with known correct chunks, and answer faithfulness given correct context. They have to be separate, because they fail differently and the fixes are unrelated. If recall is low the answer is chunking, the query rewrite or the reranker. If recall is high and faithfulness is low the answer is the generation prompt. A single end-to-end score tells you something is wrong and nothing about where, and you will spend a week finding out.

Getting the retrieval set is the part people treat as impossible and is not. Take fifty real questions, find the correct chunk for each by hand, and you have an eval that catches almost every regression. Two hours of work. The faithfulness set is the same fifty questions with the correct chunks supplied, scored on whether every claim in the answer is supported by them, which a judge model can do once you have agreed with it on a handful graded by hand.

Build ingestion, retrieval with permission filtering, and the retrieval eval first, because retrieval quality and access control are where this system succeeds or fails. Generation is comparatively easy once the right chunks arrive, and it is the part you can most safely change later.

What you would cut#

Expect to be asked, and have the list ready with a reason on each line. Cut the conversational interface and ship one question at a time, which removes the query rewrite and most of the state. Cut the reranker for the first version and take the loss in precision, because you can add it in an afternoon once the retrieval eval exists to prove it helped. Cut incremental ingestion and rebuild the whole index nightly, since 16 hours of embedding at 19 dollars is affordable and a rebuild has no invalidation bugs. Cut images, tables and anything that is not text.

Two things do not get cut, and saying which is as informative as the cuts. Permission filtering, because retrofitting it means re-indexing everything and, in the meantime, shipping a system that leaks. And the retrieval eval, because without it every later change is a guess, including all the ones you just deferred.

The last minute is worth spending on what you would watch after launch. The rate at which the system declines to answer is the single most useful number: a rising decline rate usually means the index has gone stale or ingestion is silently failing, and it moves before anyone complains. Alongside it, whether people open the cited source, which tells you whether they trust the answer, and a weekly read of twenty real traces, which tells you what they are actually asking, which is never what you assumed.