Probability & statistics interview questions
20 questions across 6 sections, in the order you would learn them. Every answer here is one that works — but an answer you have read is not an answer you can give, so use Test yourself and say it out loud first.
Probability, and reading an experiment honestly
Read the section that teaches thisIf this comes up in an interview
Your new prompt scores 86 percent against 82 percent on a 50-case eval. Ship it?
Not on that evidence. The margin of error on 50 samples is around ten points, so the difference is inside the noise. Two identical systems scored on 50 cases each differ by four points or more about 62 per cent of the time. Look at which specific cases changed, and if the direction seems real, enlarge the eval set before claiming an improvement.
Then they push further
Both were scored on the same 50 cases. Does that change your answer?
It makes the comparison much stronger, so analyse it that way: ignore the totals and count only the cases where the two disagree. If they disagree on five and the new prompt wins three, that is a coin flip. With five disagreements even a clean sweep gives a two-sided p of 0.0625, so nothing this eval could have returned would clear 0.05.
How many cases would you need?
Ask for disagreements rather than cases. Six disagreements all falling the same way is the smallest result that reaches significance in a sign test. For an unpaired comparison, the margin of error shrinks with the square root of n, so separating a genuine four-point difference needs on the order of a thousand cases per side, which is why paired comparison is worth the trouble.
What gets marked up. A passable answer says the sample is too small. A strong one gives a number for how often noise alone produces that gap, and moves to the paired analysis without being prompted.
A test for a condition affecting 1 in 1000 people is 99 percent accurate. You test positive. What is the chance you have it?
About 9 per cent. Take 100,000 people: 100 have the condition and 99 of them test positive, while 99,900 do not and 1 per cent of those, 999 people, test positive anyway. That is 1,098 positives of which 99 are real, so 99 divided by 1,098, about 0.09.
Then they push further
Why is the answer so far from 99 percent?
Because the healthy group is a thousand times larger, so even a 1 per cent error rate on it produces ten times more positives than the ill group can supply. The test did not change; the base rate dominates. At a prevalence of 1 per cent the same test gives exactly 50 per cent, and at 10 per cent it gives 92 per cent.
You test positive a second time, with an independent test. Now what?
Work in odds. Prior odds are 1 to 999, and each positive multiplies by the likelihood ratio, which is sensitivity over false positive rate, 0.99 over 0.01, so 99. After one positive the odds are 0.099, about 9 per cent; after two they are 9.81, about 91 per cent.
Where does this show up in your work?
Any classifier for a rare event. A fraud or abuse model with a 1 per cent false positive rate on a 0.1 per cent base rate sends an analyst ten false alarms per real case, so the queue gets ignored. Precision at the operating point is this same calculation, and it is what users actually experience.
What gets marked up. Reasoning in counts of people out loud is the clearest route under pressure and is marked up over reciting the formula. Saying 99 per cent, or failing to notice that 99 per cent accurate hides two separate rates, is the failure.
Why is accuracy misleading on imbalanced data?
Because the base rate dominates. A model that always predicts the majority class scores very well while being useless, and even an accurate classifier for a rare condition produces mostly false positives. Precision, recall and the cost of each error type are the honest measures.
Why report p95 latency rather than the mean?
Latency is long-tailed, so the mean is pulled around by a few slow requests and hides them at the same time. The high percentile is what users actually experience when things go wrong.
Estimation, likelihood and uncertainty
Read the section that teaches thisIf this comes up in an interview
What is maximum likelihood estimation?
Writing the probability of the data you actually observed as a function of the parameters, then choosing the parameters that make it largest. The data is fixed and the parameter is varied, which is the opposite of how the same formula is read as a probability.
Then they push further
Show it on something concrete.
The mean of a normal. The log-likelihood, dropping terms that do not involve the mean, is minus the sum of squared distances from it. Differentiating and setting to zero says the deviations must cancel, which happens only at the arithmetic average. So the sample mean is the MLE, and minimising squared error is maximum likelihood under normal noise.
Is the maximum likelihood estimate of the variance correct?
It is biased low. It measures spread around the sample mean rather than the true mean, and the sample mean sits in the middle of that same data, so the distances are too small by a factor of n minus one over n. On samples of five, a true variance of 4 comes out at about 3.2. Dividing by n minus one corrects it.
Where does the choice of loss come in?
The loss is the noise model. Normal noise gives squared error, a Bernoulli label gives cross-entropy, counts give Poisson loss, and heavier tails give absolute error. Choosing a loss is asserting how the data is noisy.
What gets marked up. Strong answers derive at least one estimator rather than defining the term, and name the squared-error connection without prompting.
Why is cross-entropy the loss for classification?
It is the negative log-likelihood of the correct labels under the model's predicted probabilities. Minimising it is maximum likelihood estimation, and it punishes confident wrong predictions heavily, which is the desired behaviour.
How would you put a confidence interval on an eval score?
Bootstrap: resample the per-example results with replacement many times, recompute the score each time, and take the 2.5th and 97.5th percentiles. It needs no distributional assumptions.
What does it mean for a model to be calibrated?
Its predicted probabilities match observed frequencies: predictions made at 80 percent are right about 80 percent of the time. Most deep networks are overconfident, which matters whenever a probability drives a decision.
Entropy, cross-entropy, KL and perplexity
Read the section that teaches thisIf this comes up in an interview
What is perplexity?
The exponential of the average cross-entropy per token. It reads as an average branching factor: a perplexity of 8 means the model is as uncertain as if it were choosing uniformly among 8 tokens at each step. Lower means less surprised by the text.
Then they push further
One model reports 12 and another reports 9. Is the second better?
Only if they share a tokenizer and were measured on the same text. Perplexity is per token, so a vocabulary that splits text into fewer, longer tokens carries more information per token and scores worse for no reason connected to quality.
How would you compare two models with different tokenizers?
Convert to bits per byte or bits per character: total negative log probability over the text, divided by the number of bytes rather than the number of tokens. Bytes are a property of the text, not a modelling choice.
Perplexity improved and your eval got worse. What happened?
They measure different things. Perplexity is fit to held-out text of a similar kind, and a model can predict such text better while doing your task worse, particularly after training on data that resembles the perplexity set more than it resembles production traffic. The task eval is the one that decides.
What gets marked up. Strong answers give the exponential-of-cross-entropy definition, the branching factor reading, and the tokenizer caveat without being prompted.
What is KL divergence?
The average extra surprise from using distribution q when the truth is p: cross-entropy minus entropy. It is zero when the two match and grows as they diverge, and it is measured in bits or nats rather than in any unit of distance.
Then they push further
Why is it not a distance?
It is asymmetric, so KL(p || q) and KL(q || p) are different numbers, and it does not obey the triangle inequality. The direction is a modelling choice: one order makes an approximation spread out to cover everything, the other makes it concentrate on a single mode.
Where would you meet KL divergence in an LLM pipeline?
As the drift penalty in preference training, keeping the tuned model near the one it started from. As the distillation loss against a teacher's full output distribution. And in monitoring, comparing the distribution of live inputs against the one the model was validated on.
Why is cross-entropy the thing being minimised rather than entropy?
Entropy is a property of the data and cannot be changed by the model. Cross-entropy is what the model's own distribution costs, so it is the only part that training can move, and the gap between the two is the KL divergence the model is actually reducing.
Probability by simulation
Read the section that teaches thisIf this comes up in an interview
A test is 99% sensitive with a 5% false positive rate for a condition with 1% prevalence. What is the chance a positive result is correct?
About 17%. Out of 10,000 people, 100 are sick and 99 test positive; of 9,900 healthy people, 495 test positive. So 99 of 594 positives are real.
Then they push further
How does this apply to machine learning?
Any classifier for a rare event has the same problem: a small false positive rate applied to a huge number of negatives produces more false alarms than true detections. Precision at the operating point is what users experience.
What gets marked up. Working with counts of people out loud, rather than recalling Bayes' formula, is the clearest way to get this right under pressure.
What does the central limit theorem say, and where does it break?
The average of many independent draws is approximately normal whatever the shape of the individual draws, centred on their mean, with a standard deviation equal to theirs divided by the square root of the sample size. It breaks when the draws are not independent, when the underlying distribution has no finite variance, and in practice long before that whenever the data is badly skewed: on exponential data, averages of 200 still have measurable skew and a lower tail 4 per cent short of the normal one.
Then they push further
Is thirty enough?
It depends entirely on the skew, which is why the rule is a rule of thumb and not a theorem. Simulate your own data: standardise the averages, check the 2.5th and 97.5th percentiles against minus and plus 1.96, and raise n until they agree. On revenue per user, where one customer in a thousand is worth a hundred times the rest, thirty is nowhere near enough.
You estimated a probability by simulation and got 0.167. How many digits should you report?
As many as the Monte Carlo margin allows, and no more. The estimate is a proportion over n trials, so the margin is 1.96 times the square root of p times one minus p over n. At 100,000 trials that is about 0.002, so two decimal places. Going to four decimals needs ten million trials, because the error falls with the square root of the work.
The distributions you will actually meet
Read the section that teaches thisIf this comes up in an interview
Why report p95 or p99 latency rather than the mean?
Latency is heavy-tailed. The mean is pulled up by a few very slow requests while describing few real ones, and it hides the tail entirely. High percentiles describe the experience of your unluckiest users, which is where problems show.
Then they push further
A request fans out to 10 backend calls in parallel and waits for all. How does the tail affect it?
It waits for the slowest of 10. If each call is slow 1% of the time, the request is slow about 10% of the time, so a backend's p99 becomes the request's p90. Fan-out magnifies tails.
What gets marked up. Marked up for the fan-out argument, which shows real systems experience.
I hand you a quantity. How do you decide which distribution to model it with?
By what generated it rather than by looking at a histogram. One trial with two outcomes is Bernoulli, and a count of successes out of a fixed n is binomial. A count of events in a window with no natural maximum is Poisson. The time until the next event is exponential. A measurement built from many small independent contributions, or any average, is normal. A quantity that is a product of effects rather than a sum — latency, file size, income — is log-normal or heavier.
Then they push further
You model support tickets per hour as Poisson and the variance is four times the mean. What does that tell you?
That the arrivals are not independent. A Poisson has variance equal to its mean by construction, so anything above that is overdispersion: tickets arrive in bursts, usually because one incident generates many. A Poisson model will badly understate the worst hours, which are the ones staffing has to cover, and a negative binomial is the usual replacement.
Why is the exponential the odd one out?
It is memoryless: having waited three minutes leaves the distribution of the remaining wait exactly as it was at the start. That is a strong claim and it fails for most real queues, where a request that is already slow is stuck on something and will stay slow, so retry and timeout policies built on it tend to be wrong.
What gets marked up. Strong answers reason from the generating process. Naming distributions from the shape of a histogram is the weaker answer and gets probed.
Hypothesis tests and intervals, practically
Read the section that teaches thisIf this comes up in an interview
What does a p-value of 0.03 mean?
If there were no real effect, data at least this extreme would occur about 3% of the time. It does not mean there is a 97% chance the effect is real, and it says nothing about the effect's size.
Then they push further
You test 20 metrics and one has p = 0.03. Should you trust it?
Not much. With 20 tests at the 5% level you expect about one false positive by chance. Correct for multiple comparisons or treat it as a hypothesis to confirm with new data.
What gets marked up. Precise wording matters here. Candidates who say 'the probability the null is true' are marked down.
How many users do you need for an A/B test?
It depends on four things and nothing else: the baseline rate, the smallest lift worth shipping, the false positive rate you accept and the power you want. For a 3.0 per cent baseline and a 0.4 point lift at 5 per cent and 80 per cent, it is about 30,000 per arm. The term that dominates is the squared difference on the denominator, so halving the lift you want to detect multiplies the sample by four.
Then they push further
You only have 5,000 per arm. What do you tell the team?
That the test has about 20 per cent power for that effect, so four times in five it will come back not significant even if the change genuinely works, and the times it does fire the measured effect will be exaggerated. Either pick a bigger effect to chase, move to a more sensitive metric, or accept that this experiment cannot answer the question.
How would you check any of this without trusting a formula?
Simulate. Generate data with the effect you care about, run the actual test you plan to run, and count how often it fires. That also catches the cases the formula does not cover, such as a skewed revenue metric or a test applied to correlated observations.
Your dashboard has twenty metrics and one moved with p = 0.03. What do you do?
Treat it as nothing. With twenty independent metrics at a 5 per cent threshold, the chance at least one fires when nothing has changed is one minus 0.95 to the twentieth, about 64 per cent, and the average number of false winners is exactly one. Name the primary metric before the experiment and treat everything else as a hypothesis for the next one.
These are the questions for one topic. The whole question bank covers every topic on the site.