Asked about The simplest case: bandits and exploration
Our ranker orders items purely by measured click-through rate. What goes wrong?
An answer that works
New items have no measurements, so they are never shown, so they never acquire any. The ranker is a greedy agent locked onto whatever was popular when it was first switched on, and the catalogue goes stale from the bottom up. The fix is to rank on an optimistic estimate rather than the measured mean, or to sample from the uncertainty, which gives new items a chance proportional to how little is known about them.
Then they push further
How would you size the exploration?
By what it costs. Simulate against your own rates: a small epsilon looks careful and is the worst option early, because the estimates it commits to are noise. An uncertainty-driven method needs no schedule, which is why it is the usual choice when you cannot afford to tune one.
Click rates drift over the year. What breaks?
Every estimator here assumes a fixed rate, so evidence from six months ago counts as much as evidence from today and the agent stops re-checking. Discount old data: an exponentially weighted mean, or decay the Beta counts towards the prior each day.
What gets marked up
Strong answers name the cold-start feedback loop rather than only saying more exploration is needed, and connect it to the locked-on failure measured in this section.
Reading an answer is not being able to give one. The section that teaches this is The simplest case: bandits and exploration, 4 hours of Reinforcement learning.
Learn this properlyAlso asked about Reinforcement learning