Distributed systems interview questions
9 questions across 4 sections, in the order you would learn them. Every answer here is one that works — but an answer you have read is not an answer you can give, so use Test yourself and say it out loud first.
The classic fundamentals
Read the section that teaches thisIf this comes up in an interview
How would you scale a read-heavy service?
Cache first, since it is the cheapest win, with an expiry chosen from how stale the data may acceptably be. Then read replicas, accepting replication lag. Sharding only when writes are the limit, since it makes cross-shard queries expensive and the shard key is hard to change later.
What is a cache stampede and how do you prevent it?
Many requests miss the same key simultaneously and all recompute the expensive value at once. Prevent it with a lock so only one recomputes while the others wait, or by staggering expiry times so keys do not all expire together.
Why do queue handlers need to be idempotent?
Queues generally guarantee at-least-once delivery, so a message can be delivered twice after a failure or a timeout. If the handler is not idempotent, that duplicate does the work twice.
Explain eventual consistency.
Replicas converge over time rather than immediately, so a read straight after a write may return the old value. Most systems accept this in exchange for availability under partition. Whether it is acceptable depends on the data: fine for a comment count, not for an account balance.
Working through the classic designs
Read the section that teaches thisIf this comes up in an interview
How would you design a rate limiter?
Token bucket: tokens refill at a fixed rate, each request spends one, and bursts are allowed up to the bucket size. Distributed, it needs shared state in a fast store with atomic check-and-decrement to avoid races between servers.
Fan-out on write or fan-out on read for a news feed?
Both. Fan-out on write makes reads fast at the cost of expensive writes, which fails for accounts with millions of followers. Those use fan-out on read. The hybrid is the standard answer.
Back-of-envelope numbers
Read the section that teaches thisIf this comes up in an interview
Estimate the storage needed for a year of chat history for 10 million daily users.
Assume about 20 messages per user per day at roughly 1 KB each including metadata: 200 million messages a day, about 73 billion a year, around 73 TB, roughly 220 TB with three replicas before indexes. Then say which assumption the answer is most sensitive to.
Your service calls eight internal dependencies. Each promises 99.9 per cent. What can you promise?
Not 99.9. Availability multiplies, so 0.999 to the eighth is about 99.2 per cent, which is around 4,200 minutes of downtime a year. To promise more than my dependencies I have to stop depending on all of them for every request: make the optional ones degrade rather than fail, cache what can be served stale, and add redundancy where the copies genuinely fail independently.
Then they push further
One of the eight is at 99 per cent. Where do you spend your effort?
On that one, and the arithmetic says so rather than instinct: with one dependency at 99 per cent the chain cannot exceed 99 per cent however good the others are. Fixing it or removing it from the critical path dominates any improvement elsewhere.
Why does adding a third redundant copy help so little?
Because after two copies that dependency is no longer the limit. Two independent copies at 99 per cent give 99.99 per cent, at which point the other services dominate the total, and improving something that is not the bottleneck does nothing.
What gets marked up. Strong answers multiply rather than quote the weakest link, and name degradation as the route to being more available than your dependencies.
Timeouts, retries and exactly-once, in practice
Read the section that teaches thisIf this comes up in an interview
How do you make a payment operation safe to retry?
Use an idempotency key generated by the client per payment attempt. The server stores the key and the outcome atomically with the charge, and returns the stored outcome for a repeated key. Combined with timeouts and backoff, retries are then harmless.
Then they push further
Can a message queue give you exactly-once processing?
Not end to end. Delivery is at least once in practice because acknowledgements can be lost. Some systems offer exactly-once within their own boundaries, but side effects outside them still need idempotent handling.
How do you save to the database and publish an event reliably?
The outbox pattern: write the event in the same transaction as the data change, and publish from the outbox table in a separate process, accepting possible duplicates.
What gets marked up. Strong answers combine idempotency keys, at-least-once delivery and the outbox pattern, and reject exactly-once claims.
These are the questions for one topic. The whole question bank covers every topic on the site.