Recommendation systems

Recommenders are among the largest machine learning systems in production anywhere, and the central constraint is one you have already met in retrieval: you cannot score a million items for every request. This section covers the two-stage shape that follows — cheap candidate generation, then expensive ranking — the signals you can match on, and what to do about a user or an item with no history at all. It also covers the uncomfortable part, that the system trains on what it previously recommended, so offline metrics can rise while the experience narrows.

What you have about this user or item      What can actually score it
years of their clicks                      collaborative: people like them
three clicks, from today                   their session, plus popularity
nothing at all, new account                popularity, by locale and device
item live for a year                       collaborative: people who took it
item uploaded ten minutes ago              content: its own features
item nobody has ever been shown            exploration, or it stays invisible

Two stages#

You cannot score a million items per request. Candidate generation narrows to a few hundred cheaply; ranking scores those precisely. The same two-stage shape as retrieval in a RAG system, for the same reason.

millions of items
      |  candidate generation, cheap and broad
      v
  ~500 candidates
      |  ranking, expensive and precise
      v
  20 shown

The reason is arithmetic rather than taste. Suppose the ranking model takes 200 microseconds per item, which is fast for anything with real features in it. Scoring two million items is 400 seconds for one request. Scoring five hundred is a tenth of a second, and that fits in a page load. There is no model that is both accurate enough to rank with and cheap enough to run over the whole catalogue, so you need two models with different jobs.

stage         items in   items out   budget   what it may use
retrieval     2,000,000  ~500        10 ms    one vector per item, ANN index
ranking       500        ~50         50 ms    full features, gradient boosting
re-ranking    50         20          5 ms     diversity, freshness, rules

The two stages fail differently and that difference decides what you work on. Retrieval fails by omission: a good item never reaches the ranker and no amount of ranking improvement can recover it, so retrieval is measured by recall — did the set of five hundred contain the things this user would have liked. Ranking fails by ordering: the right items were there and the wrong one went first, so ranking is measured by position-aware metrics. If your feed is dull, find out which of the two it is before touching either, because tuning the ranker on candidates that never contained anything interesting is a month spent reordering the same twenty items.

Collaborative filtering#

The collaborative signal is that people who behaved alike in the past will behave alike next. It needs nothing about what the items are — no titles, no genres, no descriptions — which is its defining strength: it discovers that two items belong together because the same people take both, not because somebody tagged them the same.

The item-to-item form is the one worth being able to write out. Build a matrix of users against items, treat each item as its column, and measure how similar two columns are. Recommending then means summing the similarities of everything the user already took.

import numpy as np

# Rows are users, columns are items. A one means that user took that item.
R = np.array([
    [1, 1, 0, 0, 0],
    [1, 1, 1, 0, 0],
    [0, 1, 1, 0, 0],
    [0, 0, 0, 1, 1],
    [0, 0, 0, 1, 1],
], dtype=float)

norms = np.linalg.norm(R, axis=0)
S = (R.T @ R) / np.outer(norms, norms)      # cosine similarity between items
np.fill_diagonal(S, 0)                      # an item is not its own recommendation

def recommend(user, k=1):
    scores = S @ R[user]                    # sum similarity to what they took
    scores[R[user] > 0] = -np.inf           # never recommend what they have
    return np.argsort(-scores)[:k], scores

picks, scores = recommend(0)
print(np.round(scores, 2))   # [-inf -inf 1.32 0.   0.  ]
print(picks)                 # [2]  -- item 2, which users 2 and 3 took alongside item 1

Three things it fails at, all of them structural rather than fixable by tuning. It cannot score an item nobody has taken, which is every item on its first day. It is biased towards the popular, because a column with ten thousand ones overlaps with everything and a column with four ones overlaps with almost nothing, so the tail stays invisible even when it is exactly right for someone. And it has no idea why, so when the product team asks for an explanation next to each recommendation there is nothing honest to write.

Content-based filtering#

The content signal is that this item resembles things the user already liked, where resemblance is computed from the item itself: its text, its images, its category, its creator, its price. The user is represented as a summary of the items they took, and scoring a new item means comparing it to that summary. An embedding model does this well for anything with text or images attached.

What it buys you is the entire cold-item problem solved. An item uploaded ten minutes ago has features, so it has an embedding, so it can be retrieved. That is not a small thing: for a news site, a marketplace or a video platform, most of what matters was created this week.

What it fails at is everything collaborative filtering is for. It recommends more of the same, because more of the same is what similarity means, so a user who read three articles about interest rates gets a fourth and never the one thing outside that they would actually have loved. It is capped by the quality of the features, and item metadata in the real world is entered by whoever uploaded it. And it cannot find the non-obvious pairing — the two products that have nothing in common except that the same people buy both — which is exactly the recommendation that feels like magic and is worth the most.

                       Collaborative              Content
needs                  interaction history        item features
new item               cannot score it            fine, day one
new user               cannot score them          fine, after one action
long tail              buried by popularity       reachable
surprise               finds non-obvious pairs    more of the same
explainable            not really                 yes, by feature
degrades when          the catalogue turns over   metadata is poor

Cold start#

A new user has no history and a new item has no interactions, so collaborative filtering has nothing to work with. There are four cases and they need different answers, which is why one paragraph about cold start is never enough.

• New user. Fall back to popularity conditioned on whatever you do know: country, language, device, referrer. Then switch to their session as soon as it exists, because two clicks in the current session is a stronger signal than a demographic prior. Asking them to pick three interests at sign-up works, and costs you some of them at the door.
• New item. Retrieve it by its content embedding, and give it a deliberate budget of impressions so it acquires the interaction history it needs. Without the budget it is invisible forever: no impressions means no clicks, which means no collaborative signal, which means no impressions.
• New system, no data at all. Ship content-based retrieval plus popularity and start logging. Matrix factorisation on three weeks of interactions is worse than sorting by popularity, and the logs are the asset you are actually building.
• The returning user who has been away a year. Technically warm, practically cold: their taste has moved and the last thing they clicked is about a job they no longer have. Weight recent interactions much more heavily than old ones, and treat a long gap as a partial reset.

The mechanism that makes the new-item case work is shrinkage. An item with two clicks from five impressions does not have a forty percent click rate; it has almost no information, and a greedy ranker that believes the raw ratio will put it at the top of everyone's feed on the strength of two clicks. Pull the estimate towards the catalogue average, with the pull weakening as evidence accumulates.

def smoothed_rate(clicks, impressions, prior_rate, strength=200):
    """Shrink towards the catalogue average. strength is how many impressions
    of evidence it takes before the item's own data dominates the prior."""
    return (clicks + prior_rate * strength) / (impressions + strength)

prior = 0.04                                  # the catalogue-wide click rate
for imps, clicks in [(5, 2), (200, 8), (5000, 150), (5000, 400)]:
    raw = clicks / imps
    print(imps, clicks, round(raw, 3), round(smoothed_rate(clicks, imps, prior), 4))

# 5    2   raw 0.4    smoothed 0.0488   two clicks prove nothing
# 200  8   raw 0.04   smoothed 0.04     average, and known to be average
# 5000 150 raw 0.03   smoothed 0.0304   genuinely below average
# 5000 400 raw 0.08   smoothed 0.0785   genuinely good, and allowed to say so

Feedback loops#

It is worth watching that happen rather than taking it on trust. Below, two hundred items have a fixed true click rate that the system never observes. One run always shows the current best estimates; the other gives two of its ten slots to items chosen at random. Everything else is identical.

import numpy as np

def simulate(explore_slots, rounds=400, n_items=200, slots=10, seed=0):
    rng = np.random.default_rng(seed)
    quality = rng.beta(2, 20, n_items)          # true click rates, never observed
    clicks, shown = np.zeros(n_items), np.zeros(n_items)
    for _ in range(rounds):
        estimate = (clicks + 1) / (shown + 25)  # smoothed, as above
        picks = list(np.argsort(-estimate)[: slots - explore_slots])
        if explore_slots:
            picks += list(rng.choice(n_items, explore_slots, replace=False))
        picks = np.array(picks)
        shown[picks] += 1
        clicks[picks] += rng.random(len(picks)) < quality[picks]
    return quality, shown, clicks

for explore in (0, 2):
    quality, shown, clicks = simulate(explore)
    top = np.argsort(-shown)[:10]
    print(
        "explore", explore,
        "| distinct items shown", int((shown > 0).sum()),
        "| mean true rate of the 10 most-shown", round(quality[top].mean(), 4),
        "| clicks", int(clicks.sum()),
    )

# explore 0 | distinct 182 | mean true rate of the 10 most-shown 0.1086 | clicks 481
# explore 2 | distinct 195 | mean true rate of the 10 most-shown 0.1562 | clicks 643
# The best ten items in the catalogue average 0.2302, so neither run found them.

Offline metrics mislead#

Offline you can only measure whether you would have predicted what the user actually did, under the old system. A genuinely better recommender shows different things, which offline evaluation scores as wrong. This is why recommender work depends on live experiments more than most machine learning.

# The old system only ever showed this user items 0 to 49.
exposed = set(range(50))
would_like = {3, 7, 41, 120, 155}          # unobservable: what they actually enjoy
logged = would_like & exposed              # {3, 7, 41}: all the offline test can see

def offline_recall(recs, k=5):
    return len(set(recs[:k]) & logged) / len(logged)

old_model = [3, 7, 41, 12, 19]
new_model = [3, 120, 155, 7, 88]           # surfaces two items they would love
print(round(offline_recall(old_model), 2), round(offline_recall(new_model), 2))
# 1.0 0.67

That does not make offline evaluation useless, and treating it as useless is the opposite mistake. It is reliable for the things that do not depend on what was shown: whether a change broke something, whether retrieval recall against known-relevant items went up, whether latency regressed, whether a feature has predictive value at all. Use it as a filter that kills bad ideas cheaply. It is unreliable for the size of an improvement, for anything involving items the old system rarely showed, and for the ordering of two models that are close — which is precisely when someone wants an answer from it.

• Offline predicts, roughly: this change is broken, this change does nothing, retrieval now covers items it used to miss.
• Offline does not predict: how much engagement moves, whether the feed feels repetitive, whether the new items get discovered, what happens in week three when the model has retrained on its own output.
• The last one is the one that surprises people. Every metric can improve in a two-week test and the system still degrade over a quarter, because the training data is now the product of the thing being measured.

You have the two-stage shape. This is what goes inside each stage. Retrieval is usually a two-tower model, one network for users and one for items, which lets every item embedding be precomputed and turns retrieval into nearest-neighbour search — the same architecture as dense retrieval in a RAG system. Ranking then scores the survivors with much richer features. Measurement matters as much as the model: ranking metrics care where a correct item appeared and not merely whether it did, and optimising clicks alone reliably produces a system nobody enjoys using.

Two-tower retrieval#

One network embeds users, another embeds items, and relevance is the similarity between the two. Item embeddings can be precomputed, so retrieval becomes nearest-neighbour search over millions of items. This is the same architecture as dense retrieval in a RAG system.

The reason it must be two separate towers is the whole design. A single model taking the user and the item together can express far more — how this user reacts to this price, at this hour — but it has to be run once per item, which is the million-item problem the two-stage shape exists to avoid. Keeping the towers apart forces the score to be a dot product of two vectors computed independently, and a dot product is what an approximate nearest-neighbour index can search in sublinear time. You give up the interaction terms in exchange for being able to search the catalogue at all, and then you buy the interaction terms back in the ranking stage where there are only five hundred items left.

import numpy as np

def in_batch_softmax_loss(user_vecs, item_vecs, temperature=0.05):
    """Each row is a user and the item they actually took. Every other item in
    the batch is a negative, which is why a large batch matters here."""
    u = user_vecs / np.linalg.norm(user_vecs, axis=1, keepdims=True)
    v = item_vecs / np.linalg.norm(item_vecs, axis=1, keepdims=True)
    logits = (u @ v.T) / temperature            # batch x batch
    logits -= logits.max(axis=1, keepdims=True)  # stability, changes nothing else
    probs = np.exp(logits) / np.exp(logits).sum(axis=1, keepdims=True)
    return float(-np.log(np.diag(probs)).mean())  # the diagonal is the true pair

rng = np.random.default_rng(0)
u = rng.normal(size=(8, 16))
matched = u + 0.1 * rng.normal(size=(8, 16))     # items close to their user
random_items = rng.normal(size=(8, 16))
print(round(in_batch_softmax_loss(u, matched), 3),
      round(in_batch_softmax_loss(u, random_items), 3))
# 0.0 8.699  -- the true pair is separated from its seven in-batch negatives
#              so cleanly that the loss rounds to zero; random pairings do not

Two operational facts follow from precomputing item vectors. Item embeddings go stale: the tower is retrained, and until you re-embed the catalogue and rebuild the index, the vectors in the index came from a different model than the one embedding the user, and the scores are meaningless. So retraining the retrieval model is a rebuild of the whole index, usually nightly. And the user tower has to run at request time, which is why it is kept small and why its features are ones you can look up in single-digit milliseconds.

The ranking stage#

Retrieval narrows to a few hundred candidates. Ranking scores each with far richer features: the user's recent history, the item's context, time of day, interactions between them. It can afford to be expensive because it runs on few items.

Richer means the features that could not be split into two independent towers. The count of times this user has taken something from this creator. The gap since they last saw this item. Whether the price is above or below what they usually pay. Position of the candidate in the retrieval list, which is itself informative. Gradient boosting handles these well and remains the default for anything that is not video or image heavy, because the features are tabular and there are not many candidates to score.

Ranking metrics#

All four metrics below take the same input — a ranked list, and the set of items that turned out to be relevant — and disagree about what matters. Write them out once and the disagreements become obvious.

import numpy as np

def precision_at_k(recs, relevant, k=10):
    """Of what you showed, how much was worth showing."""
    return len(set(recs[:k]) & relevant) / k

def recall_at_k(recs, relevant, k=10):
    """Of what was worth showing, how much you found."""
    return len(set(recs[:k]) & relevant) / len(relevant)

def mrr(recs, relevant):
    """How far the user scrolls before the first good thing."""
    for i, r in enumerate(recs):
        if r in relevant:
            return 1 / (i + 1)
    return 0.0

def ndcg_at_k(recs, relevant, k=10):
    """Discounted by position, normalised by the best possible ordering."""
    gains = [1 / np.log2(i + 2) for i, r in enumerate(recs[:k]) if r in relevant]
    ideal = [1 / np.log2(i + 2) for i in range(min(k, len(relevant)))]
    return sum(gains) / sum(ideal)

relevant = {"a", "b", "c"}
front = ["a", "b", "c", "x", "y", "z", "w", "v", "u", "t"]
back = ["x", "y", "z", "w", "v", "u", "t", "a", "b", "c"]

for name, recs in [("good order", front), ("same items, at the bottom", back)]:
    print(f"{name:28}",
          "P@10", round(precision_at_k(recs, relevant), 2),
          "R@10", round(recall_at_k(recs, relevant), 2),
          "MRR", round(mrr(recs, relevant), 2),
          "NDCG@10", round(ndcg_at_k(recs, relevant), 3))

# good order                   P@10 0.3 R@10 1.0 MRR 1.0  NDCG@10 1.0
# same items, at the bottom    P@10 0.3 R@10 1.0 MRR 0.12 NDCG@10 0.425

• Precision at k is the right measure when the slot count is fixed and expensive: five results on a phone screen, three items in an email. It is a poor measure when the number of relevant items is smaller than k, because the ceiling is then below one and the number is uninterpretable on its own.
• Recall at k is the right measure for the retrieval stage, where the job is to not lose anything and order does not matter yet. It is the wrong measure for the final list, where it says nothing about what the user sees first.
• MRR is the right measure when there is one correct answer and the user stops at it: a search for a specific film, a lookup. It ignores everything after the first hit, so it is the wrong measure for a feed.
• NDCG is the right measure for a ranked list with several relevant items of varying value, which is most feeds. It takes graded relevance, so a purchase can count more than a click. It is harder to explain to a product manager than the other three, which is a real cost.

Beyond clicks#

In practice that means predicting several things and combining them, rather than predicting one thing well. Each head predicts a probability — click, a long dwell, a save, a complaint — and the score is a weighted sum. The weights are a product decision written down as numbers, which is uncomfortable for everyone involved and considerably better than the alternative of pretending the objective is neutral.

def score(p):
    """One weight per predicted outcome. Negative weights are the point."""
    return (
        1.0 * p["click"]
        + 3.0 * p["long_dwell"]
        + 5.0 * p["save"]
        - 20.0 * p["hide"]
        - 50.0 * p["report"]
    )

clickbait = {"click": 0.30, "long_dwell": 0.02, "save": 0.00, "hide": 0.08, "report": 0.01}
solid = {"click": 0.12, "long_dwell": 0.09, "save": 0.04, "hide": 0.00, "report": 0.00}
print(round(score(clickbait), 3), round(score(solid), 3))
# -1.74 0.59

The other correction worth making is position bias. Users click the first result more because it is first, not because it is better, so a model trained on raw clicks learns to reproduce the previous ranking. The cheap fix is to include the position the item was shown at as a training feature and set it to a constant at serving time, which lets the model attribute some of the click to the position and the rest to the item.

Recommenders are the largest application of machine learning by revenue, and the first thing worth knowing is how good the trivial approaches are. This section builds one from interaction data: splitting by time so the model cannot learn from the future, establishing a popularity baseline that is harder to beat than it sounds, then matrix factorisation, then measuring with metrics that suit a ranked list of suggestions rather than a single prediction.

Data and split#

MovieLens, a public research dataset of film ratings, is the standard place to start. Treat any rating as an interaction. Split by time: train on everything before a date and test on what each user did after it. A random split lets the model learn from the future.

There are two time splits and they answer different questions. A global cutoff picks one date and puts everything before it in training, which is what production actually looks like: you train on Monday and serve on Tuesday. Leave-one-out holds back each user's single most recent interaction, which gives every user a test case including the ones who joined last week, and is what most papers report. The global split is harder and more honest. Use it, and if you also report leave-one-out, say which is which, because the two differ by a lot and a comparison across them is meaningless.

import numpy as np

rng = np.random.default_rng(0)
n_users, n_items = 400, 150

# A synthetic catalogue with three tastes, so there is real structure to find.
user_taste = rng.integers(0, 3, n_users)
item_taste = rng.integers(0, 3, n_items)
appeal = rng.beta(1.5, 4, n_items)                     # some items are widely liked
prob = 0.5 * (user_taste[:, None] == item_taste[None, :]) * appeal[None, :]
took = rng.random((n_users, n_items)) < prob

users, items = np.nonzero(took)
when = rng.random(len(users))                          # stands in for a timestamp
train = when < 0.8                                     # the cutoff date

R = np.zeros((n_users, n_items))
R[users[train], items[train]] = 1.0                    # what the model may see
future = [set() for _ in range(n_users)]
for u, i in zip(users[~train], items[~train]):
    future[u].add(i)                                   # what it must predict

print(int(took.sum()), "interactions,", int(train.sum()), "in training")
# 2887 interactions, 2356 in training

Popularity#

def recall_at_10(score_matrix):
    """One row of scores per user. Of everything a user took after the
    cutoff, what fraction did we put in their top ten?"""
    hits, users_scored = 0.0, 0
    for u in range(n_users):
        if not future[u]:
            continue
        s = score_matrix[u].copy()
        s[R[u] > 0] = -np.inf              # exclude what they already took
        top = np.argsort(-s)[:10]
        hits += len(set(top) & future[u]) / len(future[u])
        users_scored += 1
    return hits / users_scored

popularity = np.tile(R.sum(axis=0), (n_users, 1))   # same row for everyone
print("popularity     ", round(recall_at_10(popularity), 3))
print("random         ", round(recall_at_10(rng.random((n_users, n_items))), 3))

# popularity      0.173
# random          0.069

Recommending the most popular unseen items is embarrassingly strong. Every model must be compared with it, and many published models barely beat it once it is tuned. Tuned means at least three things: exclude what the user has already taken, restrict popularity to a recent window so last year's hit does not dominate forever, and condition it on whatever segment you cheaply know, such as country. Each of those is a line of code and each moves the baseline enough to sink a model that was about to be declared a success.

Matrix factorisation#

import torch, torch.nn as nn

class MF(nn.Module):
    def __init__(self, n_users, n_items, d=64):
        super().__init__()
        self.u = nn.Embedding(n_users, d)
        self.i = nn.Embedding(n_items, d)

    def forward(self, users, items):
        return (self.u(users) * self.i(items)).sum(-1)     # a score per pair

# Train with observed pairs as positives and random unobserved items as negatives,
# using a binary cross-entropy or pairwise ranking loss.

Each user and item gets a vector; the dot product scores how well they match. Items a user interacted with get pulled towards that user's vector. Negative sampling is needed because unobserved does not mean disliked, but you have to show the model something it should rank lower.

How you draw those negatives changes the model more than most hyperparameters do, and there are three choices with different consequences. Uniform sampling over the catalogue is easy and produces negatives that are trivially easy, because a random item from two million is obviously irrelevant and the model learns nothing from ranking it last. Sampling in proportion to popularity gives harder negatives, since a popular item the user did not take is genuine evidence of a preference, and it also corrects the bias that makes popular items score highly simply for appearing often. Sampling from the items that were actually shown and not clicked is the best signal available, and it needs impression logging that most systems only add after someone asks for it.

import numpy as np

def sample_negatives(rng, taken, n_items, k, popularity=None, alpha=0.75):
    """Draw k items the user did not take. alpha=0 is uniform, 1 is fully
    popularity-proportional; 0.75 is the usual compromise."""
    if popularity is None:
        weights = np.ones(n_items)
    else:
        weights = np.power(popularity, alpha)
    weights = weights.copy()
    weights[list(taken)] = 0.0                 # never sample a true positive
    weights /= weights.sum()
    return rng.choice(n_items, size=k, replace=False, p=weights)

rng = np.random.default_rng(0)
popularity = np.array([500.0, 300, 50, 20, 5, 2])
print(sorted(int(i) for i in sample_negatives(rng, {0}, 6, 3, popularity)))
print(sorted(int(i) for i in sample_negatives(rng, {0}, 6, 3)))

# [1, 2, 3]   popularity-weighted: it drew the two next-most-popular items
# [2, 4, 5]   uniform: it drew from the tail, where the model learns least

The simplest factorisation needs no training loop at all. A truncated singular value decomposition of the interaction matrix gives exactly the same thing — a vector per user, a vector per item, scores as their dot product — in one call, which makes it the right way to see the idea working before adding machinery.

U, sv, Vt = np.linalg.svd(R, full_matrices=False)

for k in (4, 8, 16, 32):
    mf = (U[:, :k] * sv[:k]) @ Vt[:k]                  # rank-k reconstruction
    print("factors", k, " ", round(recall_at_10(mf), 3))

# factors 4    0.432
# factors 8    0.365
# factors 16   0.381
# factors 32   0.339

That last line is the reason plain SVD is a teaching tool rather than the production answer. It treats every unobserved cell as a confident zero, so it spends capacity explaining absences. The implicit-feedback methods that replace it — alternating least squares with confidence weights, or the pairwise BPR loss — all encode the same correction: an observed interaction is evidence, an unobserved one is weak evidence at best, and they must not be weighted the same.

Measure ranking#

import numpy as np

def recall_at_k(recs, relevant, k=10):
    return len(set(recs[:k]) & relevant) / min(k, len(relevant))

def ndcg_at_k(recs, relevant, k=10):
    dcg = sum(1 / np.log2(i + 2) for i, r in enumerate(recs[:k]) if r in relevant)
    ideal = sum(1 / np.log2(i + 2) for i in range(min(k, len(relevant))))
    return dcg / ideal

Report the average by segment as well as overall, because the overall number is dominated by heavy users who have the most interactions and are the easiest to predict. The segment that decides whether the system is any good is users with fewer than twenty interactions, and it is routinely twice as bad as the headline while nobody looks.

                       recall@10   share of users
all users              0.43        100%
20+ interactions       0.58         31%
5 to 19 interactions   0.34         44%
under 5 interactions   0.11         25%

Two towers#

Matrix factorisation is a two-tower model with the simplest possible towers: a lookup table. Replace each lookup with a network that takes user features and item features, and new items with no history can still be embedded from their features. That is how large systems do retrieval over millions of items, with approximate nearest neighbour search over item vectors.

A feed or a recommendation product is a standard system design question, and it has a standard shape: cut millions of candidate items down to a few hundred cheaply, rank those carefully, re-rank for the things the ranker cannot see, and do all of it inside a latency budget. This section walks that structure from the objective down to logging and experiments. The objective comes first, because watch time and long-term retention lead to genuinely different systems.

Start with the objective#

Ask what success means before drawing boxes. Watch time, purchases, long-term retention and user satisfaction lead to different systems. Ask about scale: users, items, requests per second, and how fast new items appear.

If the objective is        the system leans towards
watch time                 long items, autoplay, strong session modelling
purchases                  price and stock features, heavy re-ranking on margin
retention over months      diversity and exploration, engagement weighted down
subscriptions              breadth of catalogue surfaced, not depth of one taste
creator ecosystem health   an impression floor for new creators, by design

The shape of the answer#

request
  -> candidate generation (several sources, ~1,000 items, tens of ms)
       two-tower ANN retrieval | recent interactions | trending | followed creators
  -> ranking (rich features, a heavier model, ~1,000 -> ~100)
  -> re-ranking (diversity, freshness, business rules, remove seen items)
  -> response
  -> logging impressions, clicks, dwell, skips  ->  training data for tomorrow

Put a number on each arrow. A feed request that must return in 200 milliseconds at the ninety-ninth percentile does not have 200 milliseconds of model time in it, and being able to say where the budget goes is what separates a design answer from a diagram.

fetch user features from the cache        5 ms
user tower forward pass                   3 ms
ANN search over the item index            8 ms   -- and the other sources, parallel
fetch features for ~1,000 candidates     25 ms   -- one batched call, never 1,000
ranking model over 1,000 candidates      40 ms
re-ranking and business rules             5 ms
                                        ------
                                         86 ms   leaving room for the network,
                                                 serialisation and one retry

Re-ranking, and why it is a separate stage#

The ranker scores each item on its own, so it cannot see that the top ten are nearly the same item. Re-ranking is where the list is judged as a list: diversity, freshness, not showing the same creator four times, honouring whatever rules the business has. Maximal marginal relevance is the standard mechanism, and it is short enough to write out.

import numpy as np

def mmr(scores, similarity, k=5, lam=0.7):
    """Pick items that are good and unlike what has already been picked.
    lam=1 is pure relevance; lam=0 is pure diversity."""
    chosen = []
    remaining = list(range(len(scores)))
    while remaining and len(chosen) < k:
        best, best_value = None, -np.inf
        for i in remaining:
            penalty = max((similarity[i][j] for j in chosen), default=0.0)
            value = lam * scores[i] - (1 - lam) * penalty
            if value > best_value:
                best, best_value = i, value
        chosen.append(best)
        remaining.remove(best)
    return chosen

scores = np.array([0.90, 0.89, 0.88, 0.60, 0.55])
# The first three are near-duplicates of each other; the last two are unrelated.
similarity = np.array([
    [1.0, 0.95, 0.93, 0.05, 0.02],
    [0.95, 1.0, 0.94, 0.03, 0.04],
    [0.93, 0.94, 1.0, 0.02, 0.03],
    [0.05, 0.03, 0.02, 1.0, 0.10],
    [0.02, 0.04, 0.03, 0.10, 1.0],
])

print(mmr(scores, similarity, k=3, lam=1.0))   # [0, 1, 2]  three of the same thing
print(mmr(scores, similarity, k=3, lam=0.7))   # [0, 3, 4]  one of each

The honest tradeoff is that diversity costs engagement today and buys it back over months, and the second half of that sentence is far harder to measure than the first. A two-week test will usually show diversity losing. That is the argument for holdback groups that run for a quarter, and it is also why this decision is made by whoever owns the product rather than by whoever owns the metric.

Logging, or there is no tomorrow#

The training data for next month's model is what today's system writes down, so the logging schema is a modelling decision, not an operational detail. Log the impression, not only the click: an item shown and ignored is the negative example the ranker needs, and a system that logs only clicks can never learn what people skip.

per impression        request_id, user_id, item_id, position, timestamp
                      the model version and the ranking score it gave
                      the candidate source that produced it
                      the feature values as used, not recomputed later
per outcome           click, dwell seconds, save, share, hide, report
                      joined back on request_id and item_id

Points interviewers look for#

• Several candidate sources, so one weak source does not starve the feed.
• The ranking objective as a combination of predicted actions, such as click, long watch and share, with weights tied to the business goal.
• Logging impressions, not only clicks, so non-clicks become negatives.
• Cold start for new users and new items, with exploration.
• Feedback loops: the model trains on what it chose to show. Some exploration keeps the data honest.
• Online experiments with guardrails, and a time-based offline evaluation.