Asked about Hypothesis tests and intervals, practically
How many users do you need for an A/B test?
An answer that works
It depends on four things and nothing else: the baseline rate, the smallest lift worth shipping, the false positive rate you accept and the power you want. For a 3.0 per cent baseline and a 0.4 point lift at 5 per cent and 80 per cent, it is about 30,000 per arm. The term that dominates is the squared difference on the denominator, so halving the lift you want to detect multiplies the sample by four.
Then they push further
You only have 5,000 per arm. What do you tell the team?
That the test has about 20 per cent power for that effect, so four times in five it will come back not significant even if the change genuinely works, and the times it does fire the measured effect will be exaggerated. Either pick a bigger effect to chase, move to a more sensitive metric, or accept that this experiment cannot answer the question.
How would you check any of this without trusting a formula?
Simulate. Generate data with the effect you care about, run the actual test you plan to run, and count how often it fires. That also catches the cases the formula does not cover, such as a skewed revenue metric or a test applied to correlated observations.
Reading an answer is not being able to give one. The section that teaches this is Hypothesis tests and intervals, practically, 4 hours of Probability & statistics.
Learn this properlyAlso asked about Probability & statistics