Reranking & hybrid search interview questions

Why use two retrieval stages?

Recall and precision want different things. The first stage is cheap and broad so the answer is definitely in the candidate set. The second is expensive and accurate and orders a short list. One stage has to compromise between them.

Why fuse search results by rank rather than score?

A BM25 score and a cosine similarity are on different scales with no meaningful conversion, so combining raw scores is arbitrary. Reciprocal rank fusion uses only position, which is comparable across any two ranked lists.

What gets marked up. Strong answers explain that keyword and vector scores are on different scales and cannot be compared directly, which is the whole reason to use ranks.

What is a cross-encoder and why not use it for everything?

It reads query and candidate together and scores the pair, which is much more accurate than comparing independently computed vectors. It also means one forward pass per candidate, so it is far too slow to run over a whole corpus.

Your RAG chatbot answers the first question well and follow-ups badly. Why, and what do you do?

Follow-ups depend on earlier turns, so embedding the raw message retrieves the wrong things. Rewrite each turn into a standalone question using the history before retrieving, and measure the change on a set of real follow-up questions.

What gets marked up. Marked up for spotting that the query lacks context and for measuring the fix rather than stacking techniques.

What is MRR and when would you use it rather than recall@k?

Mean reciprocal rank averages one over the position of the first relevant result. It rewards getting a relevant result to the very top, which matters when only the first few results are used. Recall@k measures whether the relevant results made it into the set at all.

When would you search with a hypothetical answer rather than the question, and what is the risk?

When questions and passages are written differently enough that a question embedding sits far from the passage that answers it. You ask a model for the answer it would expect and search with that, so you are comparing a passage to a passage. The risk is that the guess is specific and wrong: an invented detail becomes part of the query and pulls the search towards documents containing that detail rather than the correct one. Measure recall and MRR separately, because the usual damage is to the ranking rather than to whether the right chunk is retrieved at all.

What gets marked up. Marked up for naming the failure concretely, and for knowing the technique trades ranking for vocabulary match rather than being a free improvement.

How do you decide which query techniques to keep?

Build a small labelled set: twenty or so real turns from a chat log with the chunks that answer them. Measure recall@k and MRR for each technique added one at a time, and record the extra model calls and latency beside them. Keep the ones that move a number you care about. On most chat corpora the standalone rewrite moves recall a long way and the rest move it very little, and you cannot tell which case you are in without the measurement.