Asked about The statistics of an online experiment

Why randomise by user rather than by request?

An answer that works

Users who see both versions blur the comparison and may change behaviour because of the inconsistency. Per-user assignment gives each person one consistent experience, which is what you want to measure.

Reading an answer is not being able to give one. The section that teaches this is The statistics of an online experiment, 5 hours of Shipping and measuring.

Learn this properly

Also asked about Shipping and measuring

Your eval score improved. Is the feature better?How would you roll out a change to an AI feature?What makes A/B testing an AI feature harder than a UI change?You A/B test a new AI feature. After three days it shows a significant 4% lift. Ship it?