Asked about Running an AI service
Your prediction endpoint has a 150 ms p99 target and is missing it. Where do you start?
An answer that works
By writing the latency budget: every component on the path with its p50 and p99, measured rather than guessed. On a typical path with two feature lookups, the model forward pass is under ten per cent of the tail and the database calls are three quarters of it, so optimising the model buys almost nothing. I would also count sequential round trips before counting milliseconds, since merging two lookups into one batched call is usually the single largest win available.
Then they push further
Can you just add up the components' p99s to get the end-to-end p99?
No, and it is wrong in the safe direction. Components rarely peak together, so the sum is an upper bound. On a seven-component path with 308 ms of summed tails, simulating the components as lognormals fitted to their measured p50 and p99 gives an end-to-end p99 nearer 200 ms. Either simulate or measure end to end; never report the sum as the latency.
What gets marked up
Marked up for measuring before optimising, for knowing the sum of tails is not the tail of the sum, and for counting round trips.
Reading an answer is not being able to give one. The section that teaches this is Running an AI service, 5 hours of Deployment & serving.
Learn this properlyAlso asked about Deployment & serving