Re-ranking in RAG Pipelines: Cross-Encoders and Two-Stage Retrieval

Re-ranking in a RAG: retrieve a lot, then order well

A fast retriever returns a hundred candidates, but the right document is in seventh place, and the generator reads mostly the first ones. Two-stage re-ranking solves this with a division of labor: the first, cheap stage scans the whole collection and returns a set of candidates; the second, expensive one rereads only those candidates, scoring the question and each document together, and reorders them. The gain lies in the order of the top places, and it has a precise limit: the second stage cannot recover what the first has left out.

To make the mechanism visible you build a laboratory in which stage 1 cannot, by construction, tell two documents apart, and you see how the answer changes as the number of candidates varies. The laboratory uses only numpy and scikit-learn and no neural model; the real models appear afterwards, as a schema. To choose and evaluate the first-stage embeddings on Italian see RAG in Italian with embeddings; to combine lexical and vector search in stage 1, hybrid BM25 and vector retrieval.

Why the second stage cannot do the first

The Sentence Transformers documentation describes the scheme like this: a retrieval system produces a broad list of possible results (an order of magnitude of one hundred), then a cross-encoder-based re-ranker scores the relevance of all the candidates for that query. The bi-encoder encodes query and document separately into vectors: the document vectors are computed once and searched by proximity, so it works on millions of entries. The cross-encoder receives query and document together in a single pass and produces a score, with cross-attention between the two texts: more accurate, but one pass per pair. The same documentation observes that scoring thousands or millions of pairs would be too slow, which is why the filter of the first stage is needed.

The trade-off reads in two numbers: the cost of stage 2 grows with k, the number of candidates, not with the size of the collection; the maximum quality reachable is fixed by the share of queries for which the right document is among the top k of stage 1. Choosing k means moving along this curve, and the only honest way is to measure it on your evaluation set (see the episode on retrieval quality).

A laboratory in which stage 1 doesn't see the order

The corpus has ninety migration guides, one for each ordered pair among ten technologies: "move from angular to react" and "move from react to angular" are different documents with the same words. Stage 1 is a teaching bi-encoder: TF-IDF reduced to twenty dimensions with SVD, query and documents encoded independently, ranking by cosine. Stage 2 looks at the pair as a whole and measures the share of the query's consecutive word pairs present, in the same order, in the document. The lines to look at are stage1, which takes the top k by cosine, and stage2, which reorders only those k: its sort is stable, so on equal scores the stage 1 order remains. Neither is a neural model; it is an analogy, not a benchmark.

import numpy as np
from sklearn.decomposition import TruncatedSVD
from sklearn.feature_extraction.text import TfidfVectorizer

TECHNOLOGIES = ["angular", "react", "vue", "svelte", "django", "flask", "spring", "laravel", "rails", "express"]
# 90 guides: every ordered pair (from x to y). "from angular to react" and "from react to angular"
# use the same words, in a different order: they are two different guides.
PAIRS = [(x, y) for x in TECHNOLOGIES for y in TECHNOLOGIES if x != y]
DOCS = [f"migration guide: to move from {x} to {y} you need a plan and tests" for x, y in PAIRS]
QUERIES = [f"how to move from {x} to {y}" for x, y in PAIRS]        # the right document has the same index

# Stage 1: teaching "bi-encoder". Document and query become independent dense vectors
# (TF-IDF reduced with SVD); the document vectors are computed only once.
tfidf = TfidfVectorizer().fit(DOCS)
svd = TruncatedSVD(n_components=20, random_state=0).fit(tfidf.transform(DOCS))
doc_vectors = svd.transform(tfidf.transform(DOCS))
doc_vectors /= np.linalg.norm(doc_vectors, axis=1, keepdims=True)


def stage1(query, k):
    v = svd.transform(tfidf.transform([query]))[0]
    v /= np.linalg.norm(v)
    return np.argsort(-(doc_vectors @ v), kind="stable")[:k]


# Stage 2: teaching "cross-scorer". It looks at query and document together: share of the
# query's consecutive word pairs found, in the same order, in the document.
def word_pairs(text):
    p = text.split()
    return set(zip(p, p[1:]))


def stage2(query, indices):
    q = word_pairs(query)
    score = {i: len(q & word_pairs(DOCS[i])) / len(q) for i in indices}
    return sorted(indices, key=lambda i: -score[i])     # stable sort: on ties the stage 1 order stays


print(f"{len(DOCS)} documents, {len(QUERIES)} queries")
for k in (1, 2, 5, 10, 20):
    r1 = r_cand = r2 = 0
    for i, q in enumerate(QUERIES):
        candidates = list(stage1(q, k))
        r1 += candidates[0] == i
        r_cand += i in candidates
        r2 += stage2(q, candidates)[0] == i
    n = len(QUERIES)
    print(f"k={k:2d}: right one first at stage 1 {r1 / n:.2f} | right one among candidates {r_cand / n:.2f}"
          f" | right one first after stage 2 {r2 / n:.2f} | pairs scored by stage 2: {k}")
print("for comparison: scoring all pairs would cost", len(DOCS), "evaluations per query")

# Output:
# 90 documents, 90 queries
# k= 1: right one first at stage 1 0.50 | right one among candidates 0.50 | right one first after stage 2 0.50 | pairs scored by stage 2: 1
# k= 2: right one first at stage 1 0.50 | right one among candidates 1.00 | right one first after stage 2 1.00 | pairs scored by stage 2: 2
# k= 5: right one first at stage 1 0.50 | right one among candidates 1.00 | right one first after stage 2 1.00 | pairs scored by stage 2: 5
# k=10: right one first at stage 1 0.50 | right one among candidates 1.00 | right one first after stage 2 1.00 | pairs scored by stage 2: 10
# k=20: right one first at stage 1 0.50 | right one among candidates 1.00 | right one first after stage 2 1.00 | pairs scored by stage 2: 20
# for comparison: scoring all pairs would cost 90 evaluations per query

The first stage puts the right document in first position in 50% of cases, for every k: it sees the right words but not the direction, and the two opposite guides have identical scores, so the choice between the two is a coin toss. With k = 1 the second stage cannot change anything: it has a single candidate, and when it is the wrong one it stays wrong (0.50). With k = 2 the right candidate always appears in the set (1.00) and the reordering brings it to the top in 100% of cases. Going up to 5, 10 or 20 adds no quality in this corpus and costs more evaluations: always k pairs per query, against the 90 of a full scan. That is the point: the re-ranker adjusts the order, but only within the set it receives.

In the real world the curve is not so sharp, and k must be measured. A neural bi-encoder also captures a bit of order and context, so its errors are not random as here; and a neural cross-encoder learns the signals from data instead of counting word pairs. What the laboratory isolates remains valid: two different capabilities (retrieving and ordering), a constraint between them (the second doesn't save what the first discards), a cost that depends on k.

With the real models

The scheme in Sentence Transformers has the same shape: a bi-encoder produces the document embeddings offline, util.semantic_search retrieves the top k, a CrossEncoder scores each pair with predict and sorts by score. The name cross-encoder/ms-marco-MiniLM-L6-v2 is the one the documentation uses in the predict example; the prefix indicates training on MS MARCO, and I have not verified its behavior on Italian. For Italian texts choose a re-ranker that declares multilingual support and evaluate the gain on your own set of questions before adopting it.

# Schema to adapt, not executed: it downloads models from Hugging Face.
import numpy as np
from sentence_transformers import CrossEncoder, SentenceTransformer, util

bi = SentenceTransformer("<multilingual-bi-encoder-model>")         # stage 1
ce = CrossEncoder("cross-encoder/ms-marco-MiniLM-L6-v2")            # stage 2 (name from the documentation)

doc_embeddings = bi.encode(documents, convert_to_tensor=True)       # offline, only once
query_embedding = bi.encode(query, convert_to_tensor=True)          # at every request
candidates = util.semantic_search(query_embedding, doc_embeddings, top_k=100)[0]

pairs = [(query, documents[c["corpus_id"]]) for c in candidates]    # one model pass per pair
scores = ce.predict(pairs)
best = [candidates[i] for i in np.argsort(-scores)[:5]]             # the 5 passages to give the generator

The schema is not executed because it requires downloading the models and a set of documents. Two details matter in practice. The first: documents must enter the cross-encoder at the length the model supports, so the chunking of indexing and that of re-ranking have to be thought out together. The second: the cross-encoder score serves to order the candidates of one query; it is not comparable across different queries or different models, so don't use it as an absolute threshold without having calibrated it.

When re-ranking is worth it and when it is not

It is worth it when the bottleneck is the order of the top places and not retrieval. It can be skipped in some situations:

• a small collection, in which the bi-encoder already manages to put the right document among the top places;
• a latency budget so tight that one model pass per candidate doesn't fit;
• exact-match questions (a code, an identifier), where a lexical search or a filter does better;
• retrieval that is already poor: if the right one is not among the candidates, the re-ranker has nothing to reorder.

The fourth point is the most underestimated. Before adding a second stage, measure the share of queries for which the right document is among the top k of stage 1: if it is low, the work to do is on the first stage (chunking, embeddings, hybrid search), not on the order. To decide with a statistical test whether the re-ranker really improves things, apply the method of the episode on evaluation-driven optimization; for the overall picture of RAG architectures, naive, advanced, modular.

Frequently asked questions

What is the difference between bi-encoder and cross-encoder?

The bi-encoder encodes query and document independently and compares the vectors, so the documents are precomputed. The cross-encoder reads query and document together in one pass and produces a relevance score: more accurate, but one pass per pair.

How many candidates should I pass to the re-ranker?

As many as it takes for the right document to be almost always among them, and no more. The documentation speaks of an order of magnitude of one hundred as an example; the right value for you is found by measuring the share of queries with the right document among the top k of stage 1.

Can re-ranking make results worse?

Yes, if the model is not suited to the language or the domain. That is why the comparison must be made on your evaluation set, with the same paired method used for any other change, and not by trusting the name or the performance declared elsewhere.

What you can do now

You can explain why retrieval is split into two stages, what each one does and why the second is limited by the first; you can measure the share of useful candidates as k varies before choosing a value; you know when stage 2 isn't worth its cost and where the schema with real models is. The next step is to compare, on your own data, the system with and without a re-ranker.

The same series also includes production monitoring.