Asked about The statistics of an online experiment
You A/B test a new AI feature. After three days it shows a significant 4% lift. Ship it?
An answer that works
Not yet. Check whether the planned sample size was reached, since stopping at the first significant reading inflates false positives. Check the sample ratio, since a broken split invalidates everything. Look at guardrail metrics, and whether the effect is shrinking as novelty wears off.
Then they push further
The treatment group has 3% fewer users than control. Does that matter?
Yes, if the split was meant to be equal and the difference is statistically significant. It suggests a bug or users dropping out of one arm, often the arm with the new feature, which biases every metric. Find the cause before trusting the result.
How many users would you need to detect a 1 point lift on a 10% baseline?
Roughly 16 times 0.1 times 0.9 divided by 0.01 squared, about 14,400 per group, for 80% power at 5% significance.
What gets marked up
Strong answers raise peeking, sample ratio mismatch, guardrails and novelty. Doing the sample size estimate out loud stands out.
Reading an answer is not being able to give one. The section that teaches this is The statistics of an online experiment, 5 hours of Shipping and measuring.
Learn this properly