Skip to main content
Lab Grimoire
TW EN
Coffee
LLM Internals

Rerankers: retrieve a hundred results, then put the ones that answer the question on top

A reranker reads a question together with a short list of candidates that search already found, gives each candidate a relevance score, and reorders the list. This article explains why it is more precise and much slower than an embedding model, how two-stage retrieval splits the work, why ranking matters before text reaches an LLM, and three limits.

Author
CY
Published
Updated
AI from Principles to Practice · Part 4 of 70
Rerankers: retrieve a hundred results, then put the ones that answer the question on top

Search finds plenty, but the best result is often not first

A reranker reads a question together with a short list of candidates that search has already found, gives each candidate a relevance score, and reorders the list by that score.

On a map of AI model types, it sits between embedding models and large language models. An embedding model pulls candidates quickly out of a large collection, the reranker puts those candidates in the right order, and an LLM reads the top of the list to write an answer. Whether the first page of a literature search is useful often comes down to this step.

Rerankers at a glance

Input and output: one question plus candidates in, one score per candidate out

A reranker takes question-and-candidate pairs and returns one score for each pair.

If the first search returns 100 passages, the reranker reads the question with each passage in turn and scores it, 100 times in all. Its only job is ordering: it writes no text and fetches no new documents.

Why it is more precise: reading the question and the candidate together

A reranker is a cross-encoder: it joins the question and the candidate into one input and lets a single model read both at once, so it can judge how they relate.

An embedding model is a bi-encoder. The question and each document are turned into vectors separately, and the system compares the distance between them. Document vectors can be computed ahead of time, which makes lookup fast, but the model never sees the question while it encodes a document. A cross-encoder sees both sides every time, so it can notice which words line up, a negation, or a condition attached to the question. Think of a recruiter reading each résumé against the job description instead of matching keywords pulled from résumés in advance.

The price is speed. The Sentence-BERT paper estimates that finding the most similar pair among 10,000 sentences with a BERT cross-encoder takes about 50 million computations, roughly 65 hours, while a bi-encoder does it in about 5 seconds. That is why rerankers only ever see a short list.

Two-stage retrieval: fetch fast, then rank carefully

The common setup has two stages: a fast embedding or keyword search returns a hundred or so candidates, and the reranker picks the top ten or so.

The Sentence Transformers documentation uses this pattern as its example, retrieving the top 100 hits with a bi-encoder and re-scoring them with a cross-encoder. The first stage can also be dense passage retrieval (DPR), the bi-encoder approach from Karpukhin and colleagues. In 2019, Nogueira and Cho used BERT as the second stage to re-rank passages. The step matters most right before text goes to an LLM. If the passages it receives only brush against the question, the model may build an answer from material that does not support it. With the right passages on top, the answer is better grounded.

An example: asking a literature question

Suppose you ask: "Does regular exercise improve sleep quality in older adults?"

The first stage returns many abstracts that share the right words: animal studies of exercise and sleep, reviews of insomnia, trials of other interventions. Reading the question together with each abstract, a reranker will usually move trials that actually measured sleep quality in older people toward the top, and push mechanism-only or mouse-only studies down.

Limits: cost, missed documents, and how to read the scores

There are three limits: cost, what the first stage missed, and how the scores are read.

Doubling the candidates doubles the work. A reranker can only reorder what the first stage handed over; if the right document was never retrieved, no reranker will find it, and the fix belongs in the first stage. Its scores are best used to compare candidates within one list. Some models output a number between 0 and 1 that looks like a probability, but the same score can mean different things for different questions. Before setting a fixed cutoff for "relevant enough", check it against your own data.

How it differs from its neighbours: embedding models, LLMs, and decision models

Use an embedding model to search a large collection, a reranker to order a shortlist you already have, and an LLM only when you need new text written.

A decision model also outputs scores; the difference is what gets scored. A reranker scores whatever candidates a search returned, so the list changes with every question. A decision model scores a fixed set of options defined in advance, such as sorting an abstract into one of several study types. See how decision models work and are trained, and DecisionRAG for a retrieval approach in which a decision model picks material level by level.

References

  1. https://arxiv.org/abs/1901.04085
  2. https://arxiv.org/abs/1908.10084
  3. https://arxiv.org/abs/2004.04906
  4. https://www.sbert.net/examples/cross_encoder/applications/README.html

Frequently Asked Questions

If I already use an embedding model for semantic search, do I still need a reranker?

Not always. A reranker adds compute time, and the cost grows with the number of candidates. If the top results from the embedding search already answer the question, the gain is small. A reranker earns its cost when the results are handed to an LLM to write an answer, or when people only look at the first few hits and a wrong order is expensive. It can only reorder what the first stage retrieved; anything the first stage missed stays missing.

Can a reranker's score be read as the probability that a document is relevant?

It is safer not to. Reranker scores are mainly for comparing candidates within one list. Some models output a number between 0 and 1 that looks like a probability, but the same score can mean different levels of relevance for different questions. If you want a fixed cutoff, for example treating anything below a certain score as "nothing relevant found", test that cutoff first on a set of questions whose answers you already know.

Found this useful? RSS · X · Buy me a coffee