04 - Advanced RAG: re-ranking, hybrid search, chunk strategy

The retrieval of the previous article got five questions out of fifteen wrong: the right passage was not at the top. Faced with a number like that, the temptation is to try all the techniques you read about in blogs (BM25 ranking, hybrid search, query rewriting, re-ranking with a second model) and keep the one that looks best. The risk is choosing by luck: with fifteen questions, a single answer that changes moves hit@1 by 6.7 percentage points. This article tries four interventions on the same corpus (plus a fifth, re-ranking, described without running it) and states for each one which error it should cure, how much it really cures, and when a difference of one unit is not proof. The result anticipates the conclusion: on the test corpus no "sophisticated" technique does much, because the error that remains is a vocabulary problem, and the techniques that cure it have to bring knowledge that the corpus does not have.

The listings were run twice with identical output with numpy and scikit-learn 1.9.0; the comparison with the BM25 library was done with rank-bm25 0.2.2, installed in a separate environment. No model was downloaded or run. You need the corpus.py module of the article 03 - Fundamental RAG, to be saved next to the files of this one. As there, corpus and questions stay in Italian, because the numbers depend on the exact words; the text glosses them in English where needed, and only comments and the surrounding strings are translated.

What to measure and how far to trust it

The module below is the common infrastructure. tokens splits the text into lowercase words and, on request, removes a hand-written, deliberately short list of function words (Italian articles, prepositions, pronouns and auxiliaries). The BM25 class is the ranking introduced in the next section. valuta (evaluate) takes a function that gives a score to each passage for a question and returns hit@1, hit@3, the mean reciprocal rank (MRR) and the rank of the right passage for each question; on equal scores the lowest index wins, to get a repeatable result. The file prints nothing: the results appear in the following listings.

import math
import re
from collections import Counter

import numpy as np
from corpus import PASSAGGI, DOMANDE

TOKEN = re.compile(r"(?u)\b\w\w+\b")
# Hand-written list, deliberately short: the most frequent Italian articles, prepositions, pronouns and auxiliaries.
FUNZIONALI = set("""il lo la i gli le un una uno di a da in con su per tra fra e o ma che chi si è non al allo alla ai
alle del dello della dei delle dal dalla dai nel nella nei nelle sul sulla ho hai ha mio mia miei quanto quanti quale
quali come cui ne ci sono può deve devono va vanno è""".split())


def tokens(testo, senza_funzionali=False):
    t = TOKEN.findall(testo.lower())
    return [w for w in t if w not in FUNZIONALI] if senza_funzionali else t


class BM25:
    """BM25 with the idf of the rank_bm25 library (with a floor at 0.25 times the mean idf): k1=1.5, b=0.75."""

    def __init__(self, corpus_tokens, k1=1.5, b=0.75, eps=0.25):
        self.k1, self.b = k1, b
        self.tf = [Counter(d) for d in corpus_tokens]
        self.len = np.array([len(d) for d in corpus_tokens], dtype=float)
        self.avgdl = self.len.mean()
        n = len(corpus_tokens)
        df = Counter(w for d in self.tf for w in d)
        idf = {w: math.log(n - f + 0.5) - math.log(f + 0.5) for w, f in df.items()}
        medio = sum(idf.values()) / len(idf)
        self.idf = {w: (v if v >= 0 else eps * medio) for w, v in idf.items()}

    def punteggi(self, query_tokens):
        s = np.zeros(len(self.tf))
        norma = self.k1 * (1 - self.b + self.b * self.len / self.avgdl)
        for w in query_tokens:
            f = np.array([d.get(w, 0) for d in self.tf], dtype=float)
            s += self.idf.get(w, 0.0) * f * (self.k1 + 1) / (f + norma)
        return s


def ranghi_da_punteggi(s):
    """Position (1 = best) of each passage; on ties the lowest index wins."""
    ordine = np.argsort(-s, kind="stable")
    r = np.empty(len(s), dtype=int)
    r[ordine] = np.arange(1, len(s) + 1)
    return r


def valuta(punteggi_di):
    """punteggi_di(domanda) -> one score per passage. Returns hit@1, hit@3, MRR and the rank of each question."""
    ranghi = [int(ranghi_da_punteggi(punteggi_di(d))[gold]) for d, gold in DOMANDE]
    return sum(r == 1 for r in ranghi), sum(r <= 3 for r in ranghi), float(np.mean([1 / r for r in ranghi])), ranghi

The reading rule holds for the whole article: fifteen questions are a demonstration, not an evaluation. A change of one unit in hit@1 (9 against 10) corresponds to one question and is not enough to say that a method is better; a change of four units (9 against 13) is a signal, but remains a measurement on an invented corpus. For a real corpus you need hundreds of questions, written by whoever will actually ask them, with the right passage indicated by a person.

BM25: repetition does not count linearly

The TF-IDF of the previous article rewards words that repeat in proportion: a passage that says «copie» (copies) twice has a double score for that word, even if the second repetition adds no relevance. BM25 (a classic ranking function of text search) corrects two things: the repetition of a word saturates (with k₁ = 1.5 the contribution grows less and less) and a passage longer than the average is penalized as a function of b = 0.75. The class in the module uses the same definition of idf as the rank_bm25 library, including the floor at 0.25 times the mean idf for words that appear in more than half of the passages.

The listing computes the scores twice, by hand and with BM25Okapi, and prints the maximum gap over all questions; then it evaluates retrieval with and without function words. If the hand-written implementation were wrong, the gap would not be of the order of rounding. The second round answers a question left open: does removing «una» (a), «per» (for) and «con» (with) cure the first defect of the previous article, the hotel one?

import numpy as np
from rank_bm25 import BM25Okapi
from corpus import PASSAGGI, DOMANDE
from metriche import BM25, tokens, valuta

testi = [t for _, _, t in PASSAGGI]
print(f"{'configuration':>34} {'hit@1':>6} {'hit@3':>6} {'MRR':>6}")
for senza in (False, True):
    corpus = [tokens(t, senza) for t in testi]
    mio = BM25(corpus)
    lib = BM25Okapi(corpus)
    # check: the hand-written scores coincide with the library's on all questions
    scarto = max(np.abs(mio.punteggi(tokens(d, senza)) - lib.get_scores(tokens(d, senza))).max() for d, _ in DOMANDE)
    h1, h3, mrr, ranghi = valuta(lambda d: mio.punteggi(tokens(d, senza)))
    nome = "BM25, without function words" if senza else "BM25, all words"
    print(f"{nome:>34} {h1:>6} {h3:>6} {mrr:>6.3f}   (gap from rank_bm25: {scarto:.1e})")
    print("   ranks of the right passage:", ranghi)

# Output:
#                      configuration  hit@1  hit@3    MRR
#                    BM25, all words      9     14  0.772   (gap from rank_bm25: 6.2e-15)
#    ranks of the right passage: [1, 2, 1, 2, 2, 1, 1, 1, 1, 1, 3, 1, 2, 1, 4]
#       BM25, without function words      9     13  0.746   (gap from rank_bm25: 8.9e-16)
#    ranks of the right passage: [1, 2, 1, 9, 2, 1, 1, 1, 1, 1, 12, 1, 2, 1, 2]

The gap from the library is 10^-15: the hand-written BM25 is correct. With all words hit@3 rises from 13 (TF-IDF) to 14 and the hotel passage goes from rank 4 to 2 (and stays at 2 even without function words, so it was BM25 that cured it and not the removal), but hit@1 stays 9 (against 10 for TF-IDF, a difference of one question), so there is no demonstrable gain. Removing the function words has an effect I did not expect and that the experiment contradicts: the result gets worse (MRR 0.746 against 0.772), because the right passages of questions 4 and 11 (rank from 2 to 9 and from 3 to 12) were found mostly thanks to «con» (with) and «che» (that), words that had nothing to do with the answer. The old score was right by a lucky accident, and the removal took it away without adding synonyms. A passage that is saved by the wrong words is a fragile retrieval, not a better one.

Hybrid search: fusing two rankings

Hybrid search combines a lexical ranking (BM25) with a dense one (here the 12-dimension LSA of the previous article, with no neural model). The idea is that the two make different mistakes: the lexical one does not see synonyms but is precise on rare words, the dense one sees some closeness but can confuse specific terms. The two ways to fuse them are the mean of normalized scores and the Reciprocal Rank Fusion of Cormack, Clarke and Büttcher (2009): the score of a passage is the sum of 1/(k + rank) over all the rankings. The authors write that k = 60 was fixed in a pilot investigation and that the choice was not critical. The formula uses only ranks and not scores, so it does not require making different scales comparable; the value 60 is their setting, not a law.

The listing evaluates five methods on the same fifteen questions and prints, for each, the rank of the right passage for every question: looking at the ranks per question and not only at the mean is the way to see whether two methods get the same questions wrong.

import numpy as np
from sklearn.decomposition import TruncatedSVD
from sklearn.feature_extraction.text import TfidfVectorizer
from corpus import PASSAGGI, DOMANDE
from metriche import BM25, tokens, valuta, ranghi_da_punteggi

testi = [t for _, _, t in PASSAGGI]
bm25 = BM25([tokens(t, True) for t in testi])
vec = TfidfVectorizer(tokenizer=lambda t: tokens(t, True), lowercase=False, token_pattern=None)
X = vec.fit_transform(testi)
svd = TruncatedSVD(n_components=12, random_state=0).fit(X)
P = svd.transform(X)
P /= np.linalg.norm(P, axis=1, keepdims=True)


def lessicale(d):
    return bm25.punteggi(tokens(d, True))


def denso(d):
    q = svd.transform(vec.transform([d]))[0]
    return P @ q / (np.linalg.norm(q) + 1e-12)


def rrf(d, k=60):
    """Reciprocal Rank Fusion: sum of 1/(k + rank) over all the rankings to fuse."""
    return sum(1.0 / (k + ranghi_da_punteggi(f(d))) for f in (lessicale, denso))


def pesata(d, alfa=0.5):
    """Score fusion: each score normalized between 0 and 1 over the passages, then a weighted mean."""
    def norma(s):
        return (s - s.min()) / (s.max() - s.min() + 1e-12)
    return alfa * norma(lessicale(d)) + (1 - alfa) * norma(denso(d))


print(f"{'method':>26} {'hit@1':>6} {'hit@3':>6} {'MRR':>6}")
for nome, f in (("BM25", lessicale), ("dense (LSA, 12 dim.)", denso), ("RRF (k = 60)", rrf),
                ("weighted mean, alpha = 0.5", pesata), ("weighted mean, alpha = 0.8", lambda d: pesata(d, 0.8))):
    h1, h3, mrr, ranghi = valuta(f)
    print(f"{nome:>26} {h1:>6} {h3:>6} {mrr:>6.3f}   ranks: {ranghi}")

# Output:
#                     method  hit@1  hit@3    MRR
#                       BM25      9     13  0.746   ranks: [1, 2, 1, 9, 2, 1, 1, 1, 1, 1, 12, 1, 2, 1, 2]
#       dense (LSA, 12 dim.)      9     14  0.772   ranks: [1, 2, 1, 2, 1, 1, 1, 1, 1, 1, 12, 1, 2, 2, 2]
#               RRF (k = 60)      9     14  0.761   ranks: [1, 2, 1, 3, 1, 1, 1, 1, 1, 1, 12, 1, 2, 2, 2]
# weighted mean, alpha = 0.5      9     14  0.772   ranks: [1, 2, 1, 2, 2, 1, 1, 1, 1, 1, 12, 1, 2, 1, 2]
# weighted mean, alpha = 0.8      9     14  0.772   ranks: [1, 2, 1, 2, 2, 1, 1, 1, 1, 1, 12, 1, 2, 1, 2]

No fusion beats the dense method alone: hit@1 stays 9 for all, and the MRR goes from 0.746 (BM25) to 0.772 (dense, weighted mean) with RRF at 0.761. The differences are at most one question out of fifteen (hit@3: 13 against 14) and 0.026 in MRR. The cause can be read in the ranks: on question 11 (the «backup» that the corpus calls «copie») all methods put the right passage at rank 12, because neither BM25 nor the LSA on 18 passages knows the synonym, and a fusion of two rankings that are wrong together is wrong together.

Hybrid search helps when the two rankings have different errors; here the dense one corrects the lexical one on questions 4 (the lost cellphone) and 5 (how long backups are kept) (ranks from 9 to 2 and from 2 to 1), but makes it worse on 14 (reimbursement with the car) (from 1 to 2), and the RRF inherits the first two gains only in part (question 4: rank 3). With a pretrained embedding model the two rankings would probably be more different, but this article did not run one and does not claim it.

Rewriting the question, and HyDE

If the error lies in the vocabulary, you can change the question before searching. Rewriting (query rewriting) asks a language model to rephrase the question, or to add terms; HyDE, proposed by Gao, Ma, Lin and Callan (2022), asks the model to write a hypothetical document that answers the question, and searches for passages close to that document, which may contain false details but carries the kind of words a true passage would contain. Both techniques require a call to a model for every question: cost, latency and a model that may add wrong terms.

No model is run here. The listing simulates the rewriting with a dictionary of synonyms that I wrote by hand after seeing the failures, and applies it to the questions: it adds to the question the words of the document (for example «telefono» (phone) for «cellulare» (cellphone) and «copie» (copies) for «backup»); other entries pair «albergo» (hotel) with «pernottamento» (overnight stay) and «respinta» (rejected) with «scartata» (discarded). The number that comes out is not an estimate of the effectiveness of any model: it is a constructed upper bound, which says how much a perfect rewriting would be worth on the right words.

from corpus import PASSAGGI, DOMANDE
from metriche import BM25, tokens, valuta

testi = [t for _, _, t in PASSAGGI]
bm25 = BM25([tokens(t, True) for t in testi])

# Query rewriting: SIMULATED with a dictionary written by hand after seeing the failures.
# It measures no model: it only shows what happens if the question carries the words of the document.
SINONIMI = {
    "cellulare": ["telefono"], "codici": ["secondo", "fattore"], "backup": ["copie", "ripristinare"],
    "albergo": ["pernottamento"], "tetto": ["fino", "rimborsato"], "portatile": ["computer"],
    "automobile": ["auto"], "dentista": ["visite", "mediche"], "respinta": ["scartata"],
    "cancellato": ["ripristino"], "conservate": ["conservano"],
}


def espandi(domanda):
    t = tokens(domanda, True)
    return t + [s for w in t for s in SINONIMI.get(w, [])]


for nome, q in (("original question", lambda d: tokens(d, True)), ("question with synonyms", espandi)):
    h1, h3, mrr, ranghi = valuta(lambda d: bm25.punteggi(q(d)))
    print(f"{nome:>22}: hit@1 {h1}  hit@3 {h3}  MRR {mrr:.3f}  ranks {ranghi}")

# Output:
#      original question: hit@1 9  hit@3 13  MRR 0.746  ranks [1, 2, 1, 9, 2, 1, 1, 1, 1, 1, 12, 1, 2, 1, 2]
# question with synonyms: hit@1 13  hit@3 15  MRR 0.933  ranks [1, 1, 1, 2, 1, 1, 1, 1, 1, 1, 1, 1, 2, 1, 1]

With the dictionary, hit@1 goes from 9 to 13 and the MRR from 0.746 to 0.933; only questions 4 and 13 (who approves a credit note) stay at rank 2. It is the largest gain of the article, and for that very reason it is the least credible: the dictionary was built by looking at the answers. A rewriting done by a real model can add the right terms and can also add wrong ones, and its effectiveness has to be measured on questions it has not seen. The message is another one: when retrieval fails because of vocabulary, the lever is to bring the words of the document into the question (or the question into the space of the documents, as embeddings do), not to refine the ranking.

Small units, broad answer

The choice of the unit to index has a trade-off: a small unit (a sentence) is more similar to a question, a large unit has the context to answer. Parent document retrieval takes the best of both: sentences are indexed, but when a sentence wins, the passage that contains it is returned. The listing divides the 18 passages into 32 sentences (15.8 words on average against 28.2), indexes the sentences with BM25, gives each passage the score of its best sentence, and compares the result with the index on whole passages.

import re
import numpy as np
from corpus import PASSAGGI, DOMANDE
from metriche import BM25, tokens, valuta

testi = [t for _, _, t in PASSAGGI]

# Index of small units (sentences), each with a reference to the passage that contains it.
frasi, padre = [], []
for i, t in enumerate(testi):
    for f in re.split(r"(?<=[.;])\s+", t):
        frasi.append(f)
        padre.append(i)
padre = np.array(padre)
print(f"{len(testi)} passages, {len(frasi)} sentences; words per passage {np.mean([len(t.split()) for t in testi]):.1f}, per sentence {np.mean([len(f.split()) for f in frasi]):.1f}")

bm_passaggi = BM25([tokens(t, True) for t in testi])
bm_frasi = BM25([tokens(f, True) for f in frasi])


def via_frasi(domanda):
    s = bm_frasi.punteggi(tokens(domanda, True))
    return np.array([s[padre == i].max() for i in range(len(testi))])     # the passage is worth as much as its best sentence


for nome, f in (("index on passages", lambda d: bm_passaggi.punteggi(tokens(d, True))),
                ("index on sentences, returns the passage", via_frasi)):
    h1, h3, mrr, ranghi = valuta(f)
    print(f"{nome:>46}: hit@1 {h1}  hit@3 {h3}  MRR {mrr:.3f}")

# Output:
# 18 passages, 32 sentences; words per passage 28.2, per sentence 15.8
#                              index on passages: hit@1 9  hit@3 13  MRR 0.746
#        index on sentences, returns the passage: hit@1 10  hit@3 13  MRR 0.780

The index on sentences improves slightly (hit@1 from 9 to 10, MRR from 0.746 to 0.780): one more question, so below the threshold of a signal. The expected advantage is nonetheless real in principle: the sentence at the top concentrates the comparison on the words that matter, and the whole passage stays available to build the prompt. The price is an index with almost twice the entries and a map to maintain between sentences and parents.

Re-ranking with a cross-encoder

Re-ranking is a second stage: the first stage retrieves many candidates with an inexpensive method (BM25, fusion, vectors), the second orders them with a more expensive model that reads the question and each candidate together (a cross-encoder). The reason for the two passes is cost, which Reimers and Gurevych quantify for the worst case: reading every pair with a single network over 10,000 sentences requires 49,995,000 inferences, about 65 hours on a V100 GPU. Re-ranking 20 candidates per question requires 20 inferences. This article does not run any cross-encoder, because a model would have to be downloaded; the block is a schema to adapt, and the syntax of CrossEncoder and of predict (which receives a list of pairs) follows the Sentence Transformers documentation consulted today.

# SCHEMA TO ADAPT, NOT EXECUTED: it requires downloading a re-ranking model.
from sentence_transformers import CrossEncoder


def riordina(domanda, candidati, modello, k=3):
    """`candidati` are the passages of the first stage (for example the top 20 of the fusion)."""
    coppie = [(domanda, c) for c in candidati]
    punteggi = modello.predict(coppie)             # one inference per (question, passage) pair
    ordine = sorted(range(len(candidati)), key=lambda i: -punteggi[i])
    return [candidati[i] for i in ordine[:k]]


modello = CrossEncoder("NAME-OF-THE-RERANKING-MODEL")   # to be chosen and evaluated on your own data

The schema does three things: it builds the pairs, asks for the scores with predict (one inference per pair) and sorts. The name of the model is deliberately a placeholder, because the model has to be chosen and evaluated on your own data and your own language: a re-ranking model trained on English questions guarantees nothing for an Italian corpus. I measured no gain and I do not claim one.

What you can do now

You can measure retrieval with hit@k and MRR and read the ranks per question, you can verify a hand-written BM25 against a library, you can fuse two rankings with RRF and recognize when the fusion cannot help, and you can tell a gain from a one-question difference. You can also describe rewriting, HyDE and cross-encoder re-ranking with their cost, without having run them. The next article moves from retrieval to action: 05 - AI Agents: ReAct, tool-use, planning. If you first need basic retrieval, start from 03 - Fundamental RAG; the index is on the series page.

Later in the series you will also find 08 - On-prem inference with Ollama, vLLM, llama.cpp, which estimates the memory and the running time of the model that receives the retrieved context.