Ranking models and how to measure them

You have the two-stage shape. This is what goes inside each stage. Retrieval is usually a two-tower model, one network for users and one for items, which lets every item embedding be precomputed and turns retrieval into nearest-neighbour search — the same architecture as dense retrieval in a RAG system. Ranking then scores the survivors with much richer features. Measurement matters as much as the model: ranking metrics care where a correct item appeared and not merely whether it did, and optimising clicks alone reliably produces a system nobody enjoys using.

Two-tower retrieval#

One network embeds users, another embeds items, and relevance is the similarity between the two. Item embeddings can be precomputed, so retrieval becomes nearest-neighbour search over millions of items. This is the same architecture as dense retrieval in a RAG system.

The reason it must be two separate towers is the whole design. A single model taking the user and the item together can express far more — how this user reacts to this price, at this hour — but it has to be run once per item, which is the million-item problem the two-stage shape exists to avoid. Keeping the towers apart forces the score to be a dot product of two vectors computed independently, and a dot product is what an approximate nearest-neighbour index can search in sublinear time. You give up the interaction terms in exchange for being able to search the catalogue at all, and then you buy the interaction terms back in the ranking stage where there are only five hundred items left.

import numpy as np

def in_batch_softmax_loss(user_vecs, item_vecs, temperature=0.05):
    """Each row is a user and the item they actually took. Every other item in
    the batch is a negative, which is why a large batch matters here."""
    u = user_vecs / np.linalg.norm(user_vecs, axis=1, keepdims=True)
    v = item_vecs / np.linalg.norm(item_vecs, axis=1, keepdims=True)
    logits = (u @ v.T) / temperature            # batch x batch
    logits -= logits.max(axis=1, keepdims=True)  # stability, changes nothing else
    probs = np.exp(logits) / np.exp(logits).sum(axis=1, keepdims=True)
    return float(-np.log(np.diag(probs)).mean())  # the diagonal is the true pair

rng = np.random.default_rng(0)
u = rng.normal(size=(8, 16))
matched = u + 0.1 * rng.normal(size=(8, 16))     # items close to their user
random_items = rng.normal(size=(8, 16))
print(round(in_batch_softmax_loss(u, matched), 3),
      round(in_batch_softmax_loss(u, random_items), 3))
# 0.0 8.699  -- the true pair is separated from its seven in-batch negatives
#              so cleanly that the loss rounds to zero; random pairings do not

Two operational facts follow from precomputing item vectors. Item embeddings go stale: the tower is retrained, and until you re-embed the catalogue and rebuild the index, the vectors in the index came from a different model than the one embedding the user, and the scores are meaningless. So retraining the retrieval model is a rebuild of the whole index, usually nightly. And the user tower has to run at request time, which is why it is kept small and why its features are ones you can look up in single-digit milliseconds.

The ranking stage#

Retrieval narrows to a few hundred candidates. Ranking scores each with far richer features: the user's recent history, the item's context, time of day, interactions between them. It can afford to be expensive because it runs on few items.

Richer means the features that could not be split into two independent towers. The count of times this user has taken something from this creator. The gap since they last saw this item. Whether the price is above or below what they usually pay. Position of the candidate in the retrieval list, which is itself informative. Gradient boosting handles these well and remains the default for anything that is not video or image heavy, because the features are tabular and there are not many candidates to score.

Ranking metrics#

All four metrics below take the same input — a ranked list, and the set of items that turned out to be relevant — and disagree about what matters. Write them out once and the disagreements become obvious.

import numpy as np

def precision_at_k(recs, relevant, k=10):
    """Of what you showed, how much was worth showing."""
    return len(set(recs[:k]) & relevant) / k

def recall_at_k(recs, relevant, k=10):
    """Of what was worth showing, how much you found."""
    return len(set(recs[:k]) & relevant) / len(relevant)

def mrr(recs, relevant):
    """How far the user scrolls before the first good thing."""
    for i, r in enumerate(recs):
        if r in relevant:
            return 1 / (i + 1)
    return 0.0

def ndcg_at_k(recs, relevant, k=10):
    """Discounted by position, normalised by the best possible ordering."""
    gains = [1 / np.log2(i + 2) for i, r in enumerate(recs[:k]) if r in relevant]
    ideal = [1 / np.log2(i + 2) for i in range(min(k, len(relevant)))]
    return sum(gains) / sum(ideal)

relevant = {"a", "b", "c"}
front = ["a", "b", "c", "x", "y", "z", "w", "v", "u", "t"]
back = ["x", "y", "z", "w", "v", "u", "t", "a", "b", "c"]

for name, recs in [("good order", front), ("same items, at the bottom", back)]:
    print(f"{name:28}",
          "P@10", round(precision_at_k(recs, relevant), 2),
          "R@10", round(recall_at_k(recs, relevant), 2),
          "MRR", round(mrr(recs, relevant), 2),
          "NDCG@10", round(ndcg_at_k(recs, relevant), 3))

# good order                   P@10 0.3 R@10 1.0 MRR 1.0  NDCG@10 1.0
# same items, at the bottom    P@10 0.3 R@10 1.0 MRR 0.12 NDCG@10 0.425

• Precision at k is the right measure when the slot count is fixed and expensive: five results on a phone screen, three items in an email. It is a poor measure when the number of relevant items is smaller than k, because the ceiling is then below one and the number is uninterpretable on its own.
• Recall at k is the right measure for the retrieval stage, where the job is to not lose anything and order does not matter yet. It is the wrong measure for the final list, where it says nothing about what the user sees first.
• MRR is the right measure when there is one correct answer and the user stops at it: a search for a specific film, a lookup. It ignores everything after the first hit, so it is the wrong measure for a feed.
• NDCG is the right measure for a ranked list with several relevant items of varying value, which is most feeds. It takes graded relevance, so a purchase can count more than a click. It is harder to explain to a product manager than the other three, which is a real cost.

Beyond clicks#

In practice that means predicting several things and combining them, rather than predicting one thing well. Each head predicts a probability — click, a long dwell, a save, a complaint — and the score is a weighted sum. The weights are a product decision written down as numbers, which is uncomfortable for everyone involved and considerably better than the alternative of pretending the objective is neutral.

def score(p):
    """One weight per predicted outcome. Negative weights are the point."""
    return (
        1.0 * p["click"]
        + 3.0 * p["long_dwell"]
        + 5.0 * p["save"]
        - 20.0 * p["hide"]
        - 50.0 * p["report"]
    )

clickbait = {"click": 0.30, "long_dwell": 0.02, "save": 0.00, "hide": 0.08, "report": 0.01}
solid = {"click": 0.12, "long_dwell": 0.09, "save": 0.04, "hide": 0.00, "report": 0.00}
print(round(score(clickbait), 3), round(score(solid), 3))
# -1.74 0.59

The other correction worth making is position bias. Users click the first result more because it is first, not because it is better, so a model trained on raw clicks learns to reproduce the previous ranking. The cheap fix is to include the position the item was shown at as a training feature and set it to a constant at serving time, which lets the model attribute some of the click to the position and the rest to the item.