Dense retrieval or keyword search
Should retrieval match on embeddings, on BM25 keyword scoring, or on both?
The short answer
Hybrid with a reranker is the strongest default. Start with BM25 alone, because it is free and it tells you honestly how much the embeddings are adding.
Dense (embedding) retrieval
Good at
- Matching meaning across different words — questions phrased nothing like the source text.
- Multilingual retrieval, where the query and the document are not in the same language.
- Short, vague, conversational queries that share no terms with the answer.
- Being robust to paraphrase, which is how people actually ask questions.
Costs you
- Bad at exact strings: part numbers, error codes, names and identifiers all blur.
- Re-embedding the whole corpus whenever you change embedding model.
- Index memory and an embedding call on every query.
- Out-of-domain terms the embedding model never saw are represented badly and silently.
- Hard to explain a match, which makes debugging retrieval slow.
Pick this when: Natural-language questions over prose, where the wording of the query will not match the wording of the source.
Keyword search (BM25)
Good at
- Exact matches: codes, identifiers, function names, rare technical terms.
- Being cheap, instant to index, and available in every search engine and database already.
- Explainability — you can see exactly which terms matched and why.
- Working on day one with no model, no tuning and no embedding bill.
- Being a surprisingly strong baseline that many RAG systems never actually beat.
Costs you
- Vocabulary mismatch: the right document loses because it says 'salary' and the query said 'pay'.
- No sense of synonyms or paraphrase without a hand-built thesaurus.
- Sensitive to stemming, stopwords and tokenisation choices.
- Weak on short queries where every term is common.
Pick this when: Corpora full of identifiers and jargon, and as the baseline you must beat before claiming your dense retriever works.
Hybrid, then rerank
Good at
- Covering both failure modes — the exact-string miss and the paraphrase miss.
- Recall first, precision second: fetch broadly from both, then rank with a cross-encoder.
- Being the configuration that wins most published retrieval comparisons.
Costs you
- Two indexes to keep in sync and two sets of tuning knobs.
- Fusing two score scales is fiddly, and reciprocal rank fusion is a heuristic, not a solution.
- Reranking adds latency and a second model in the request path.
- More moving parts to instrument before you can tell which stage failed.
Pick this when: Retrieval quality is the bottleneck and you have measured that it is.