Reranking

Reranking is a second scoring pass over passages that a first retrieval already returned. The first pass is optimised to search a large index quickly; the second is optimised to judge relevance well over a short list. Almost every serious retrieval system runs both, and the split explains a great deal about which pages end up quoted.

Anthropic's description of the technique is compact: with large knowledge bases the initial retrieval often returns a lot of chunks — sometimes hundreds — of varying relevance and importance, and reranking is a commonly used filtering technique to ensure only the most relevant reach the model. Its stated benefit is not only accuracy: passing fewer, better passages also reduces cost and latency in the generation step, because the model processes less.

The steps

The published pipeline is four moves:

1. Perform an initial retrieval to collect the potentially relevant chunks — in Anthropic's configuration, the top 150.
2. Pass those chunks, together with the user's query, through a reranking model.
3. Score each chunk on relevance and importance to the prompt, and take the top few — in the same configuration, the top 20.
4. Pass those into the model as context to generate the final answer.

The important structural difference from the first pass is that a reranker sees the query and the passage together. A vector search compares two independently computed embeddings; a reranker reads the pair and judges it. That is far more accurate and far too slow to run over a whole corpus, which is exactly why it is the second stage and not the first.

What it is worth

Anthropic's measurements are the clearest public numbers for the effect in isolation, all expressed as top-20 retrieval failure rate, or one minus recall at 20:

• baseline embeddings: 5.7%
• contextual embeddings plus contextual BM25: 2.9%, a 49% reduction
• the same, reranked: 1.9%, a 67% reduction

So the reranker took roughly a third off what remained after the retrieval improvements. Its own summary of the whole experiment is that these benefits stack, and that the best configuration combined contextual embeddings, contextual BM25, a reranking step and 20 chunks in the prompt.

The cost side, stated honestly

Reranking is not free. Anthropic notes that it inevitably adds latency because it is an extra runtime step, even though the reranker scores all candidates in parallel, and that there is an inherent trade-off between reranking more chunks for better performance and fewer for lower latency and cost. Its recommendation is to experiment on the specific use case rather than adopting a fixed number.

For a product decision this matters more than the accuracy figure. A hundred-and-fifty-candidate rerank on every query is a real per-query bill, and the systems that answer questions about your brand are making that trade in ways you cannot see.

Why a publisher should care

Reranking is the stage that decides citation position — not merely whether your passage was found, but whether it survives the cut and lands near enough to the front to be the one paraphrased.

Two practical implications follow.

The first is that being retrievable is not the finish line. A passage can clear the first pass on topical similarity and then lose the second because a competing passage answers the question more directly. The reranker is comparing your paragraph against the query, so a paragraph that restates the question and then answers it in the next sentence is being scored on exactly the thing it was written to do.

The second is that padding hurts twice. A long, hedged passage dilutes the signal a reranker is scoring and consumes budget in the top-K that a sharper passage would have used. This is the same discipline described under chunking — one subject per section, the claim stated in the heading, the qualifier in the same sentence as the claim — arriving from a different direction.

None of this is a ranking factor you can file a request against. It is a description of the machinery, and the only lever it exposes is writing a passage that reads as a direct answer to a question somebody actually asked.