A structure for the design round
By the end: You can run a 45-minute AI design round without freezing or rambling.
- Clarifying before designing
- An order that works
- Being explicit about trade-offs
- What separates a mid answer from a senior one
New to ai system design? Working out the shape of a whole AI feature before any of it is built: where the model sits, what information reaches it and how that information is found, what is stored between one request and the next, what happens when a step fails, what a request costs, and how anyone would know if it got worse. In an interview it is done aloud at a whiteboard in under an hour. Why it matters
Forty-five minutes, a deliberately vague prompt, and no structure of your own is how this round goes wrong. This section gives you one: what to establish before designing anything, an order to build the design in, and how to talk about trade-offs as you make them. The last part matters most, because naming what a choice costs is largely what separates a senior answer from a competent one.
Do not start designing#
The most common failure is drawing boxes in the first minute. Spend five minutes establishing what you are building, for whom and at what scale. Interviewers deliberately give vague prompts to see whether you ask.
Questions worth asking in almost any AI design round:
How many users, and how many requests each?
How fresh must the data be? Seconds, hours, days?
Is latency user-facing or is this a background job?
What is the cost of a wrong answer? Annoying, or expensive?
Do we need to show where an answer came from?
What is the budget per request?That last pair matters more in AI systems than in conventional ones, because both cost and correctness are dials rather than fixed properties.
An order that works#
- 1.Clarify requirements and state the ones you are assuming.
- 2.Sketch the end-to-end flow at a high level, request to response.
- 3.Name where the model sits and what exactly it is asked to do.
- 4.Cover data: ingestion, storage, indexing, freshness.
- 5.Cover failure: what happens when the model is wrong, slow or down.
- 6.Cover cost: rough per-request maths, and the levers you would pull.
- 7.Cover evaluation: how you would know it regressed.
- 8.Say what you would build first and what you would defer.
Steps 5 to 7 are where candidates separate
Most candidates draw a reasonable architecture and stop. Failure handling, cost and evaluation are what distinguish someone who has run a system in production from someone who has read about one. Raise them yourself rather than waiting to be asked.
The numbers you are expected to produce#
Every AI design round asks for four estimates, usually without asking for them by name: requests per second, tokens per request, cost per thousand requests, and storage. None of them needs a calculator, all of them need you to have the conversion factors already in your head, and every one of them is arithmetic you should say out loud rather than announce the answer to.
Worth memorising, because you will not look them up at a whiteboard:
seconds in a day 86,400 (call it 100,000)
seconds in an 8-hour working day 28,800 (call it 30,000)
1 million requests a day ~12 per second
1 million requests a month ~0.4 per second
peak traffic 3x the daily average
English words to tokens x 1.33
a page of prose (500 words) ~665 tokens
generation speed ~40 tokens per second
one 1536-dim float32 embedding 6,144 bytes, call it 6 KBThe four estimates then fall out in order. Requests per second is the daily volume divided by 86,400, times three for peak, and the point of saying it is usually to discover the system is small: ten thousand queries a day is a third of a request per second, which means this is a quality problem and not a scaling one. Tokens per request is the prompt plus the retrieved material plus the output, and the retrieved material dominates. Cost per thousand requests is tokens times price, and stating it per thousand rather than per request keeps you away from numbers with four leading zeros. Storage is rows times bytes per row.
Say the units as you go. An estimate with the units attached survives being wrong, because the interviewer can see where the mistake is and correct one factor; an unlabelled number that is wrong just looks like guessing.
A design round, annotated, with a clock#
What follows is one round end to end. The times on the left are what a reasonable pace looks like; the lines in brackets are what the interviewer is writing on the scorecard. The prompt is deliberately ordinary, and the answer is not the only correct one.
00:00 INTERVIEWER Design a service that moderates user-generated
content with an LLM.
00:30 YOU Before I draw anything I want five numbers. How many
items a day, and what kinds - text only, or images too?
Is this pre-publish, so a user waits, or post-publish?
What is the policy: a fixed list of categories, or
something that changes weekly? And what happens to a
flagged item - blocked, queued for a human, or shadowed?
[asks before designing - the first line on every rubric]
01:30 INT 50 million daily actives, about four posts or comments
each. Text for now. Post-publish, but fast. The policy
changes often. Flagged items go to a human queue.
02:00 YOU Then let me size it. Fifty million times four is 200
million items a day. Over 86,400 seconds that's about
2,300 a second average, and I'll plan for three times
that at peak, so roughly 7,000 a second.
02:40 YOU That number decides the architecture on its own. Seven
thousand LLM calls a second is not a thing I am going to
build, so the LLM cannot be on the main path for every
item. It has to be the second stage.
[capacity arithmetic before components - the separator]Notice what the arithmetic did. It did not decorate a design that was already decided; it ruled one out. That is the move being marked, and it is why the numbers come before the diagram rather than after it.
03:00 YOU So: three tiers. A hash and blocklist check on exact
repeats and known-bad strings, in-process, microseconds.
Then a small fine-tuned classifier - a few hundred MB,
served on CPU, a couple of milliseconds an item - that
scores every item against the policy categories and
returns a confidence. Then the LLM, but only for the
band in the middle where the classifier is unsure.
04:30 INT How wide is that band?
04:40 YOU I'd tune it against the human queue's capacity rather
than picking a threshold. Say it comes out at 2 per
cent: that's 4 million LLM calls a day, 46 a second
average, 140 at peak. That I can build.
[threshold derived from a constraint, not invented]
05:30 YOU The LLM's job is narrow: given the item and the current
policy text, return a category, a confidence, and the
clause it relied on. Structured output, no free prose.
The policy lives in the prompt rather than in weights,
which is the whole reason the LLM tier exists - the
classifier needs retraining when the policy changes and
the prompt does not.
06:30 INT Why not send everything to the LLM?
06:40 YOU Cost and capacity. I'll do that number in a minute, but
roughly it's fifty times the traffic at the same unit
price. The cheap tier exists to make the expensive tier
affordable.Failure comes next, and it comes from you rather than from a prompt. The useful discipline is to walk the diagram left to right and ask what happens when each box is wrong, slow or absent, because that produces specific answers instead of a general assurance that you would add monitoring.
08:00 YOU Failure, going along the path.
Classifier wrong, false negative: harmful content is
published. That's the expensive one, so I'd bias the
threshold towards sending more to the LLM and accept the
cost, and sample published items for audit.
Classifier wrong, false positive: an innocent post is
queued. Cheap, but at 200 million items even 0.1 per
cent is 200,000 a day, which is more than a human team
can read. So false positive rate sets the queue size,
and I'd say that out loud to product.
LLM provider down: fall back to the classifier's own
decision with a conservative threshold, and mark those
items for re-review when the provider returns. Never
fail open on a moderation system.
LLM slow: it is post-publish, so it is a queue, not a
request. Latency degrades into a longer backlog rather
than a timeout, which is the main reason I chose
post-publish when I had the choice.
Prompt injection in a post: the item text is user data
and adversarial by definition. It goes in a separate
delimited block, the LLM has no tools, and its output is
parsed as an enum rather than executed.
Policy changed this morning: cached decisions from
yesterday's policy are now wrong. Cache key includes a
policy version, so a policy change invalidates cleanly.
[six failure modes, each with the design that answers it]Then cost, and here the arithmetic has to be visible. Say the token counts, say the rates you are assuming, and do the multiplication aloud. Getting a factor wrong out loud costs you nothing; presenting a number you cannot derive costs you the round.
20:00 YOU Cost. Per LLM call: the policy is about 800 tokens,
the item maybe 60, and the structured answer 40 out.
I'll assume $1 per million in and $5 per million out.
860 in x $1 / 1M = $0.00086
40 out x $5 / 1M = $0.00020
$0.00106 per call, so $1.06
per thousand calls.
4 million calls a day is about $4,240 a day, so roughly
$127,000 a month. That is too much, and the policy is
the same 800 tokens every time, so I'd cache the prefix.
Cache reads run about a tenth of input price:
800 cached x $0.10 / 1M = $0.00008
60 fresh x $1 / 1M = $0.00006
40 out x $5 / 1M = $0.00020
$0.00034 per call, $0.34 per
thousand, about $1,360 a day
and $40,800 a month.
Three times cheaper for one change, and it constrains
the design: the policy has to sit at the front of the
prompt, ahead of the item, or the prefix never matches.
[arithmetic shown, not asserted - and a lever with a
structural consequence]
22:00 INT What if that is still too much?
22:10 YOU Then I widen the classifier's confident bands, which
trades recall for money, and I'd want to see the recall
curve before agreeing to it. Or batch the queue and use
the provider's batch tier. I would not switch to a
smaller model without re-running the eval, because on
this workload a wrong decision is the expensive thing.The last five minutes are the ones most candidates never reach, and they are worth disproportionate credit because almost nobody gets there. Evaluation, then what you would build first, then what you would cut.
35:00 YOU How I'd know it regressed. Two eval sets, not one: a
labelled set per policy category with the hard cases
over-sampled, and a set of adversarial items that got
through before. Report recall and false positive rate
separately, because a single accuracy number hides the
trade that matters. Run both on every policy edit -
policy text is a deployment.
38:00 YOU Build order: the classifier and the human queue first,
with no LLM at all. That is a working moderation system,
it produces the labelled data the LLM tier needs, and it
tells me the real distribution of what people post.
The LLM tier is week four.
40:00 YOU What I'd cut if I had to ship in a month. Images. The
appeals flow - route appeals straight to a human until
there is volume to justify automating them. Per-user
reputation, which is a real win but needs history I do
not have yet. I would not cut the audit sample of
published items, because without it I have no idea what
the false negative rate is and I would be guessing.
[cuts named with reasons, and one thing refused]The rubric#
Interviewers are usually filling in a form, and the rows are more predictable than the questions. An adequate answer produces a workable architecture and defends it when challenged. A strong one arrives at the architecture from constraints, quantifies before choosing, raises its own weaknesses, and knows which parts to drop.
Adequate Strong
scoping answers questions asked asks five before drawing
architecture correct components each one justified by a
constraint that was stated
numbers "it would be expensive" arithmetic out loud, with
units and a lever
failure answers when prompted raised unprompted, one per
component, with the design
that handles it
trade-offs picks a side names the cost and the
condition that flips it
evaluation mentions monitoring an eval set, what it
measures, and what a
regression looks like
scope designs everything says what ships first and
what is cut, and whyThe ways strong candidates lose are few and specific. They design before clarifying, which produces a good system for a problem nobody asked about. They give no numbers, so every claim about cost or scale is an adjective. They cover the happy path and stop, leaving failure and evaluation unmentioned until the last five minutes when there is no time. They make choices without naming what those choices cost, which reads as not knowing that they cost anything. And they present a finished design instead of building one aloud, which is fatal in a round whose entire purpose is to watch you reason.
Running out of time is a scoring event
Forty-five minutes is enough for scoping, architecture, failure, cost, evaluation and scope only if you keep moving. Candidates who spend twenty minutes perfecting the retrieval layer never reach cost or evaluation, and the scorecard has rows for both. If you are at the halfway mark and still drawing components, stop drawing and say what happens when they fail.
State trade-offs out loud#
There is no correct answer, and the interviewer knows it. What is being assessed is whether you can name what each choice costs.
"I'd cache aggressively on the retrieval layer. That cuts cost
and latency, and it means a document updated five minutes ago
might not show. Given the freshness requirement we agreed, I
think that's the right trade. If it weren't, I'd shorten the TTL
and accept the higher cost."Numbers beat adjectives#
Saying a system will be expensive is worth little. Estimating roughly two thousand input tokens and five hundred output at a given rate, so around three cents per request and thirty thousand dollars a month at a million requests, changes the conversation entirely. Rough arithmetic done out loud reads as experience.
Where people go wrong
Designing in silence and presenting a finished answer. The round assesses reasoning, so a good design arrived at invisibly scores worse than a decent one argued out loud.
If this comes up in an interview
Design a system that answers questions over a company's internal documents.
Clarify first: corpus size, freshness, latency, whether citations are required, budget. Then ingestion and structural chunking, hybrid retrieval with reranking, generation with citations and a threshold for declining when retrieval is weak. Then failure handling, per-request cost with the levers, and a retrieval eval separate from a generation eval. Finish with what I would build first.
Then they push further
Different employees can see different documents. How does that change the design?
Permissions have to be enforced at retrieval. Store access data with each chunk and filter by the user's permissions inside the search query, never by asking the model to ignore documents. Keep the index in sync when permissions change.
How do you know it is working after launch?
Log every trace, read a sample weekly, track feedback and whether people open the cited sources, and run the eval set on every change. A rising 'I do not know' rate often means the index has gone stale.
Answers take 8 seconds and people are complaining. Where do you look?
Break the time down by stage: retrieval, reranking, time to first token and generation. Then stream the answer, cut the context, cache frequent queries, and see whether reranking or a query rewriting call can be dropped or run in parallel.
What gets marked up. Strong answers ask about scale and requirements first, then cover ingestion, permissions, retrieval, cited generation, evaluation and monitoring. Weak answers stop at embed, store, retrieve, generate.
How do you handle the model being wrong in a production system?
Assume it will be. Show sources so users can verify, make correction cheap, keep a human in the loop where being wrong is expensive, gate destructive actions behind confirmation, and log everything so a bad answer can be traced. Design for wrong answers rather than trying to eliminate them.
Then they push further
Give a concrete example of that design.
For a refund assistant: the model drafts a decision with the policy it relied on, code checks the amount against the order and the policy limit, and anything over a threshold goes to a person. Users see the policy cited, so they can check it themselves.
What gets marked up. Marked up for designing on the assumption the model will sometimes be wrong: checks in code, human review where the cost is high, and errors that are visible and recoverable.
How would you know the system had regressed?
A fixed eval set run on every change, reporting quality, cost and latency together, with a held-out split. Plus production monitoring on the same signals, and sampling real traffic back into the eval set so it keeps describing what the system is actually asked.
Build this: Run three 45-minute designs out loud against a timer: a document question-answering system, an agent that acts on a customer account, and a content moderation pipeline. Record yourself and check whether you covered failure, cost and evaluation without prompting.
Watch it explained
This is one section of AI system design, 17 hours across 3 sections. Read it in full to get the whole topic in order.
Stop 37 of 50 on the AI Engineer route
0 of 50 topics done · 0h behind you, 603h to go
What you have marked done is stored in this browser only. It does not reach a server, and clearing site data will clear it.