Two-Stage Retrieval: Recall Then Precision

Chapter 7 left two gaps open: the retriever misses documents whose wording differs from the query (gap 1), and buries good documents below the cut-off (gap 2). This chapter closes both, with two ideas that combine into the standard production architecture.

Idea one: search two ways at once

Chapter 2’s embeddings match on meaning. That is powerful and it has a specific blind spot: things with no meaning to speak of.

Consider the query Configure NVIDIA_VISIBLE_DEVICES. Dense retrieval has no semantic intuition about that identifier — it is an arbitrary string. It may well rank a document about “configuring GPU visibility settings” above the one that contains the literal variable name.

Old-fashioned keyword search — sparse retrieval, usually BM25 — has the opposite profile. It matches exact terms brilliantly and fails completely on paraphrase, because “how do I make my AI faster” and “LLM latency optimisation” share no words.

Query type	Example	Better retriever
Conceptual	“How do transformers learn?”	Dense
Exact identifier	NVIDIA_VISIBLE_DEVICES	Sparse
Named entity	“Dr. Chen’s 2024 paper”	Sparse
Error code	“What does HTTP 429 mean?”	Sparse
Paraphrase	“make AI faster” → “LLM optimisation”	Dense
Mixed	“cost of the streaming API”	Both

Real queries are the last row far more often than any other. So run both and combine.

Combining two rankings that do not speak the same language

Here is the catch. A dense search returns cosine similarities around 0.87. BM25 returns scores like 14.2. These numbers are not comparable — not by scaling, not by normalising, because their distributions differ per query.

The trick is to throw the scores away and keep the ranks.

Reciprocal Rank Fusion scores each document by where it placed in each list:

def reciprocal_rank_fusion(rankings, k=60):
    scores = defaultdict(float)
    for ranking in rankings:     # one per retriever
        for rank, doc_id in enumerate(ranking):
            scores[doc_id] += 1 / (k + rank + 1)
    return sorted(scores.items(), key=lambda x: -x[1])

A document ranked 1st by dense and 30th by sparse gets 1/61 + 1/90. A document ranked 5th by both gets 1/66 + 1/66 — and wins. Agreement across different methods beats a single strong opinion, which is exactly the behaviour you want.

The constant k (conventionally 60) damps the influence of the very top ranks, so one retriever’s confident mistake cannot dominate. RRF needs no tuning, no score calibration, and no training. It is the default for good reason.

Idea two: look properly, but only at the finalists

Now the deeper problem — the one built into Chapter 7’s bi-encoder.

The document was embedded before anyone knew what the query would be. Its vector had to anticipate every possible question. That is why it can be approximately right and precisely wrong.

A cross-encoder does the opposite. It takes the query and the document together, in one pass, and lets attention run across both. Every word of the query can interact with every word of the document. It produces one relevance score, and it is far more accurate.

	Bi-encoder (stage 1)	Cross-encoder (stage 2)
When encoded	Documents offline, once	Query and document together, per request
Sees interaction?	No	Yes
Cost per query	One lookup over millions	One model pass per candidate
Scales to	Billions of documents	Tens to low hundreds

Look at the cost row and the architecture writes itself. You cannot cross-encode a million documents per query. You can cross-encode a hundred.

The same shape, one level down

The funnel applies to the vectors themselves, not only to the candidate count. Storage and search cost scale with dimensions: a million documents at 1,536 dimensions in 4-byte floats is about 6 GB before any index overhead, and it has to be fast to search, which usually means resident in memory.

Matryoshka embeddings — the name is the nesting-doll reference — are trained so that the first N numbers of a vector are themselves a valid, if slightly worse, embedding. You truncate, and quality degrades gracefully instead of breaking.

full = model.encode(text)   # 1024 dimensions
cheap = full[:128]          # still a usable embedding

Which gives stage 1 a cheap-then-expensive split of its own:

Goal	Dimensions	Effect
Peak accuracy	1024–3072	Best quality, highest cost
Two-stage search	128, then 1024	Search wide at 128-d, re-score the top 100 at full width
Cost-sensitive	256	Roughly 12x storage saving, low single-digit quality loss
Edge / on-device	64	Fast and small; simple intents only

Search wide at 128 dimensions, re-score the survivors at full width, then rerank. Cheap-and-wide then expensive-and-narrow, nested inside itself.

The pipeline, assembled

1. Rewrite the query if it is conversational or underspecified.
2. Retrieve with dense and sparse in parallel, ~50–100 candidates each.
3. Fuse with RRF into one ranked list.
4. Rerank the top ~100 with a cross-encoder.
5. Select the top 3–10, ordered so the strongest sit first and last.
6. Generate.

Steps 2 and 3 close gap 1. Step 4 closes gap 2. Step 5 mitigates gap 3. That is the whole of Chapter 7’s failure taxonomy, addressed by one pipeline.

When to add which stage

You do not need all of this on day one. Add stages in response to measured failures:

Symptom	Add
Misses exact names, codes, identifiers	Sparse retrieval + RRF
Right document retrieved but ranked low	Cross-encoder reranking
Conversational follow-ups retrieve badly	Query rewriting
Good chunks retrieved, answer still ignores them	Fewer chunks, better ordering

What to carry forward

Two retrievers beat one because they fail differently. Fuse them by rank, not by score. Then re-score the finalists with a model that reads query and document together — the accuracy you could not afford across the whole corpus becomes affordable across a hundred candidates.

Cheap-and-wide, then expensive-and-narrow. That closes Part II. Part III asks what happens when the system stops answering and starts acting.