Back-of-envelope numbers

In a design discussion someone will ask how much traffic, how much storage, how much bandwidth, and expect an answer in under a minute. This section is the small set of arithmetic and reference numbers that lets you give one: turning users into requests per second, requests into bytes per year, and knowing roughly how long a memory read, a cross-continent round trip and a model call each take. Rounding hard and saying your assumptions out loud is the skill being tested; precision is not.

Users to load#

10 million daily users x 20 requests each = 200 million requests/day
a day is about 86,400 s, call it 100,000 for easy maths
  average = 2,000 requests/s
  peak    = 2 to 5 x average, say 10,000 requests/s

Three habits in that block are the whole technique. A day is 100,000 seconds, not 86,400, because you are estimating and the extra fifteen per cent is far inside your other error bars. Peak is a multiple of average rather than a separate calculation, and the multiple is two to five for a consumer product with one big timezone and can be ten or more for anything with a scheduled event in it. And you size for peak, because the average never took a service down.

Storage#

each chat message about 1 KB with metadata
10 million users x 20 messages x 365 days = 73 billion messages/year
  x 1 KB = 73 TB/year, before replication (x3) and indexes

The two multipliers at the end are where estimates are usually wrong, and both are worth saying unprompted. Replication is three copies by default, so the disk you buy is three times the data you have. Indexes and metadata commonly add another twenty to fifty per cent on top of that, and for anything with embeddings in it the vectors can exceed the text they were made from — a thousand-dimensional float32 vector is 4 KB, four times the 1 KB message it represents. Say which assumption the answer is most sensitive to and you have demonstrated the thing being tested.

Latency, in orders of magnitude#

read from memory                          ~100 ns
read 1 MB sequentially from memory        ~tens of microseconds
random read from an SSD                   ~tens to hundreds of microseconds
round trip within a data centre           ~0.5 ms
round trip across a continent             ~tens of ms
round trip between continents             ~100 to 200 ms
LLM time to first token                   ~hundreds of ms to seconds

The use of that table is subtraction. Given a latency budget, spend it. A 300 ms budget for a page load, minus a hundred milliseconds for a user on another continent, minus a few milliseconds per internal hop, leaves under two hundred milliseconds of real work — which a single model call will consume entirely, and which three sequential model calls cannot fit into at all. That is why a design with a chain of model calls has to stream, cache or run them in parallel, and noticing it from the arithmetic rather than from a production incident is the point.

The arithmetic of availability#

Availability multiplies along a dependency chain, and almost nobody's intuition does. If your request needs ten services and each is up 99.9 per cent of the time, you are not up 99.9 per cent of the time. You are up 0.999 to the tenth, and the difference is the gap between an afternoon of downtime a year and most of a working week.

YEAR = 365 * 24 * 60          # minutes

print("dependencies   each 99.9%    each 99.99%    downtime/year at 99.9% each")
for n in [1, 3, 5, 10, 30]:
    a3, a4 = 0.999 ** n, 0.9999 ** n
    print(f"{n:12}   {a3:9.4%}    {a4:10.4%}    {(1-a3)*YEAR:12,.0f} minutes")

print()
print("one flaky dependency at 99% inside an otherwise 99.99% chain of 10:")
chain = 0.9999 ** 9 * 0.99
print(f"  combined {chain:.4%}  ->  {(1-chain)*YEAR:,.0f} minutes a year")

print()
print("making that one redundant instead of fixing it:")
for copies in [1, 2, 3]:
    dep = 1 - 0.01 ** copies
    combined = 0.9999 ** 9 * dep
    print(f"  {copies} independent copies at 99%: dependency {dep:.4%}, "
          f"chain {combined:.4%}, {(1-combined)*YEAR:,.0f} minutes")

# dependencies   each 99.9%    each 99.99%    downtime/year at 99.9% each
#            1    99.9000%      99.9900%             526 minutes
#            3    99.7003%      99.9700%           1,575 minutes
#            5    99.5010%      99.9500%           2,623 minutes
#           10    99.0045%      99.9000%           5,232 minutes
#           30    97.0431%      99.7004%          15,541 minutes
#
# one flaky dependency at 99% inside an otherwise 99.99% chain of 10:
#   combined 98.9109%  ->  5,724 minutes a year
#
# making that one redundant instead of fixing it:
#   1 independent copies at 99%: dependency 99.0000%, chain 98.9109%, 5,724 minutes
#   2 independent copies at 99%: dependency 99.9900%, chain 99.9000%, 525 minutes
#   3 independent copies at 99%: dependency 99.9999%, chain 99.9099%, 473 minutes

Two escapes from the multiplication are worth naming. Make the dependency optional: a recommendation service that is down should produce a page without recommendations, not a 500, and a dependency you can degrade past has left the chain entirely. Or make it redundant, as the last block does — but redundancy only multiplies if the copies fail independently, and two instances in the same rack behind the same deploy pipeline do not. The correlated-failure caveat is worth saying out loud, because every real outage of a redundant system is a correlated failure.

Tail latency and fan-out#

The last piece of arithmetic is the one that explains why big systems feel slow in ways their dashboards do not show. If a request fans out to many backends and waits for all of them, its latency is the maximum of theirs, not the average, so a rare slow response becomes a common slow request.

print("a request that fans out to N shards, each with a 1-in-100 chance of being slow")
print("shards   requests hitting at least one slow shard")
for n in [1, 5, 10, 20, 100]:
    print(f"{n:6}   {1 - 0.99 ** n:38.1%}")

print()
print("hedging: send to two replicas and take whichever answers first")
for n in [10, 20, 100]:
    one, two = 1 - 0.99 ** n, 1 - (1 - 0.01 ** 2) ** n
    print(f"{n:4} shards   one copy {one:6.1%}   hedged {two:6.1%}")

# a request that fans out to N shards, each with a 1-in-100 chance of being slow
# shards   requests hitting at least one slow shard
#      1                                     1.0%
#      5                                     4.9%
#     10                                     9.6%
#     20                                    18.2%
#    100                                    63.4%
#
# hedging: send to two replicas and take whichever answers first
#   10 shards   one copy   9.6%   hedged   0.1%
#   20 shards   one copy  18.2%   hedged   0.2%
#  100 shards   one copy  63.4%   hedged   1.0%

What to keep and what to skip#

• Keep: a day is 100,000 seconds, peak is two to five times average, replication is three times, and a millisecond is a data-centre round trip.
• Keep: availability multiplies, so count the dependencies before quoting a target.
• Keep: fan-out turns a tail percentile into a common case.
• Skip: precise arithmetic. Round hard, say the assumption, and move on; nobody is checking your long division.
• Skip: memorising the full latency table to three significant figures. The orders of magnitude and their ratios are the whole of it.