01 - RAG evaluation crisis: why "it works" is not a metric

"It works" is an impression, not a measurement

A RAG system tried on twenty questions, with answers that look good, gives you an impression and no number. The impression holds while the corpus is small and the questions are the ones you thought of yourself. Then documents arrive that resemble each other, users who do not use your words, and a manager who asks how reliable it is: at that point "it works" answers nothing. To answer you need three things, and they are the same in every project.

• a set of questions chosen with a criterion, not at random and not only the easiest ones;
• for each one, the right answer or the right document, decided by a person before looking at the system's output;
• a score reported with its interval of uncertainty, not with three decimals that nobody could justify.

The items are in order of cost: the first takes an afternoon, the second weeks of work by someone who knows the domain, the third a minute of computation but requires accepting that 0.85 on a hundred questions is not 0.85. This article opens the series and builds the common ground: the ways a RAG breaks without raising errors, the four metrics that measure them, two checks that need no model, and the mathematics that says how many questions you need. The next steps are the articles on Ragas, on how to build the golden dataset and on the metrics of the retriever alone.

A note on method. The listings below use an invented corpus of ten sentences about an imaginary software product and TF-IDF lexical retrieval with scikit-learn: no model is called and no vector database is started. It is a teaching model, which shows the mechanisms with reproducible numbers; it does not measure any real system, and the printed percentages should not be read as estimates of how reliable a production RAG is.

Four ways a RAG gets it wrong without warning

A wrong RAG throws no exceptions: it returns a well-written text. That is why it is worth classifying the failures before choosing the metrics, because each metric sees one failure and ignores the others.

The retriever does not find the document. The question uses words different from those of the text: "can I get my money back" against "a refund is possible within 14 days". A lexical retriever has no word in common to exploit. An embedding retriever may cope, but it too gets it wrong when the domain is specialised. The generator, without the right passage, answers anyway.

The context is dirty. With a high top-k, chunks that are close in topic but useless get in. The model has to ignore them and does not always manage. Liu and colleagues measured that answer quality changes with the position of the information: it is often highest when it sits at the beginning or at the end of the context and drops when it sits in the middle, even in models designed for long contexts. This is not a reason to cut the context to two chunks, it is the reason to measure the order in which the retriever delivers its results.

The credible hallucination. The context is correct but incomplete, and the generator completes it in its own way: from the chunk with the price it also derives a number of users that no document reports. It is the most dangerous failure because it survives a quick read by a person. A preregistered study of commercial RAG-based legal research tools found that the LexisNexis and Thomson Reuters tools hallucinated between 17% and 33% of the time: less than general-purpose chatbots, then, but far from the zero promised by the vendors. The figure concerns those closed products and that domain, not your system; it serves as a reminder that "there is a retriever" does not mean "it does not make things up".

The corpus ages or duplicates itself. Two versions of the same document live together in the index, with different numbers, and the retriever has no reason to prefer the updated one. The following listing isolates this mechanism. It starts from ten documents and ten questions, each with a single right document, and adds for each document k copies in which the digits have changed. The vectoriser only counts words (token_pattern discards numbers), so a copy has the same score as the original; the insertion order is shuffled with a fixed seed, so no copy is favoured by position.

# b1.py - the retriever "works" on the demo and loses ground when the corpus fills up with near-duplicates
import random
import numpy as np
from sklearn.feature_extraction.text import TfidfVectorizer

BASE = {
    "pricing":      "The Pro plan costs 49 euros a month and includes 10 users.",
    "refund":       "A refund is possible within 14 days of purchase, by contacting support.",
    "export":       "Data export produces a CSV file from the Settings page.",
    "sso":          "SSO login is available only on the Enterprise plan through SAML.",
    "backup":       "Automatic backups run every night and are kept for 30 days.",
    "api":          "The API accepts at most 100 requests per minute for each key.",
    "invoices":     "Invoices can be downloaded as PDF from the Billing section of the account.",
    "password":     "To reset your password use the link sent to your registration email.",
    "integrations": "Integrations with Slack and Teams are enabled from the Integrations panel.",
    "limits":       "Each project accepts up to 500 documents uploaded in PDF format.",
}
# (question, document that contains the answer)
QUERY = [
    ("how much does the Pro plan cost per month", "pricing"),
    ("can I get my money back", "refund"),                  # no words in common: different vocabulary
    ("how do I download my data as CSV", "export"),
    ("is SAML required to log in", "sso"),
    ("how often are backups made", "backup"),
    ("how many requests per minute does the API accept", "api"),
    ("where do I download invoices as PDF", "invoices"),
    ("I forgot my password", "password"),
    ("how do I connect Slack", "integrations"),
    ("how many documents can I upload per project", "limits"),
]

def with_near_duplicates(k: int, seed: int = 7) -> dict:
    """Adds k variants per document: same text, digits changed (old versions)."""
    rng = random.Random(seed)
    corpus = dict(BASE)
    for name, text in BASE.items():
        for j in range(k):
            variant = "".join(str(rng.randint(1, 99)) if c.isdigit() else c for c in text)
            corpus[f"{name}#v{j}"] = variant
    order = list(corpus)
    rng.shuffle(order)                       # insertion order must not favour the original
    return {n: corpus[n] for n in order}

def evaluate(corpus: dict) -> tuple[float, float]:
    ids = list(corpus)
    vec = TfidfVectorizer(sublinear_tf=True, token_pattern=r"(?u)\b[^\W\d_]{2,}\b")  # words only, digits do not count
    M = vec.fit_transform(corpus.values())
    hit1 = hit3 = 0
    for question, right in QUERY:
        scores = (M @ vec.transform([question]).T).toarray().ravel()
        ranked = [i for i in np.argsort(-scores, kind="stable") if scores[i] > 0][:3]   # score 0 = no result
        top = [ids[i] for i in ranked]
        hit1 += bool(top) and top[0] == right
        hit3 += right in top
    return hit1 / len(QUERY), hit3 / len(QUERY)

for k in (0, 2, 10, 50):
    c = with_near_duplicates(k)
    h1, h3 = evaluate(c)
    print(f"documents={len(c):4d}  hit@1={h1:.0%}  hit@3={h3:.0%}")

# Output:
# documents=  10  hit@1=80%  hit@3=90%
# documents=  30  hit@1=10%  hit@3=80%
# documents= 110  hit@1=10%  hit@3=20%
# documents= 510  hit@1=0%  hit@3=0%

With ten documents the retriever misses two questions out of ten at hit@1, for two different reasons. "Can I get my money back" shares not a single word with the refund sentence: the only document with a non-zero score is the invoices one, through the generic word "can", so hit@3 stays at 90% even afterwards. In "is SAML required to log in" the function words "in" and "to" push the limits sentence above the right one, which still comes second. The rest is another matter: the right document has the same score as its copies, and the order among ties decides who rises.

With 30 documents hit@1 falls to 10% and hit@3 to 80% (the right one is among the first three, rarely first), with 110 hit@3 also falls to 20%, and with 510 the original never appears in the first three places. No universal threshold on corpus size appears here: it depends on how much the documents resemble each other, and that is precisely why it must be measured on your corpus. The countermeasure is not a better model, it is a version metadata field and a filter before retrieval.

Two checks that need no model

Before paying a judge for every answer, it is worth pulling out what can be computed in one line. Retrieval is checked with identifiers: if for each question you know which documents contain the answer, retrieval_ok verifies that at least one appears in the top k. Faithfulness has a rough but useful check for one precise case: numbers. If the answer states "25 users" and 25 is not in the context, something was invented. numbers_without_support extracts the digit sequences from the answer and removes those present in the context.

# b3.py - a minimal dashboard without a model: two free checks that catch real errors
import re
from dataclasses import dataclass

@dataclass(frozen=True)
class Case:
    id: str
    question: str
    expected_ids: frozenset     # documents that contain the answer (human label)
    retrieved_ids: tuple        # what the retriever returned, in order
    context: str                # text of the chunks passed to the generator
    answer: str                 # what the generator said

NUMBER = re.compile(r"\d+(?:[.,]\d+)?")

def retrieval_ok(c: Case, k: int = 3) -> bool:
    """Check 1: at least one expected document is among the top k."""
    return bool(c.expected_ids & set(c.retrieved_ids[:k]))

def numbers_without_support(c: Case) -> list:
    """Check 2: numbers the answer states and the context does not contain."""
    seen = set(NUMBER.findall(c.context))
    return sorted(set(NUMBER.findall(c.answer)) - seen)

CASES = [
    Case("q1", "How much does the Pro plan cost?", frozenset({"pricing"}), ("pricing", "api", "sso"),
         "The Pro plan costs 49 euros a month.",
         "The Pro plan costs 49 euros a month."),
    Case("q2", "How many users does the Pro plan include?", frozenset({"users"}), ("pricing", "api", "sso"),
         "The Pro plan costs 49 euros a month.",
         "The Pro plan costs 49 euros a month and includes up to 25 users."),
    Case("q3", "How many users does the Pro plan include?", frozenset({"users"}), ("pricing", "api", "sso"),
         "The Pro plan costs 49 euros a month.",
         "The context does not say how many users the plan includes."),
]

for c in CASES:
    print(f"{c.id}: retrieval_ok={retrieval_ok(c)}  numbers_without_support={numbers_without_support(c)}")

# Output:
# q1: retrieval_ok=True  numbers_without_support=[]
# q2: retrieval_ok=False  numbers_without_support=['25']
# q3: retrieval_ok=False  numbers_without_support=[]

Case q2 is the credible hallucination of the previous section: the retriever got it wrong (retrieval false) and the generator filled the gap with a 25 that the context does not contain, so the check flags it. Case q3 is more instructive: retrieval failed in the same way, but the answer declares that it does not have the information, and no number is invented. The two checks are independent and tell you two different things: system q3 is honest and useless, system q2 is dishonest and useless. A single score would make them indistinguishable.

The limit is obvious: the check sees only numbers. A date written in words, a wrong name or a negation slip through. For those you need a judge that breaks the answer into statements and compares them with the context, which is what the metrics described below do. Do not try to extend the check with regular expressions: every exception you add is one more false alarm.

The four metrics and the failure each one sees

Ragas, described in the paper by Es and colleagues, proposes metrics that evaluate the dimensions of a RAG separately, and its documentation gives the operational definitions. Each has a different minimum input, and that is the difference that matters when you must decide how much annotation work to require.

Read the table by the "what you need" column, not by name. Precision and recall, in the versions that ask for them (the library also contains a variant of precision without a reference, ContextPrecisionWithoutReference), use a reference answer: they cost human work. Faithfulness and relevancy do not: the system's output and a judge are enough. That is why you start from them when you do not yet have a golden dataset, and why the article on Ragas devotes space to generating synthetic data. Note also what the table does not say: none of the four checks that the answer is true. Faithfulness is 1 even for a wrong answer, if the retrieved context was wrong in the same way.

The definitions, from the Ragas documentation. Context precision is the average of the precisions at each position occupied by a relevant chunk, divided by the number of relevant chunks: a right chunk in first position weighs more than a right one in eighth. Faithfulness is the ratio between the statements of the answer supported by the context and the total statements; a score of 0.75 means that a quarter of the statements find no support. Answer relevancy takes a stranger route: it asks the model to generate a few questions (three by default) starting from the answer, computes the cosine similarity of each with the original question and averages them. An evasive answer produces generic questions, far from the original. The documentation specifies that the value is normally between 0 and 1 but, because of the cosine, this is not guaranteed.

The listing that follows shows the four calls in the current interface, ragas.metrics.collections. It was not run: it requires a client for a judge model and an embedding model, and here no model is called. The signatures were read with inspect.signature on version 0.4.3 installed in a temporary folder. Check the version you use: the interface changed between versions (imports from ragas.metrics emit a deprecation warning pointing to ragas.metrics.collections), and an example copied from a 2024 article may no longer run.

# b4.py - schema to adapt (not run): you need a client for a judge model and a key of your own
# Signatures checked with inspect.signature on ragas 0.4.3; no metric was computed.
import asyncio
from ragas.llms import llm_factory
from ragas.embeddings.base import embedding_factory
from ragas.metrics.collections import AnswerRelevancy, ContextPrecision, ContextRecall, Faithfulness

llm = llm_factory("<judge-model-name>", client=ASYNC_CLIENT_OF_YOUR_PROVIDER)
emb = embedding_factory("openai", model="<embedding-name>", client=ASYNC_CLIENT_OF_YOUR_PROVIDER)

question = "How many users does the Pro plan include?"
contexts = ["The Pro plan costs 49 euros a month and includes 10 users."]
answer = "The Pro plan includes 10 users."
reference = "The Pro plan includes 10 users."

async def main():
    faithfulness = await Faithfulness(llm=llm).ascore(
        user_input=question, response=answer, retrieved_contexts=contexts)       # no reference
    relevancy = await AnswerRelevancy(llm=llm, embeddings=emb).ascore(
        user_input=question, response=answer)                                     # no contexts
    precision = await ContextPrecision(llm=llm).ascore(
        user_input=question, reference=reference, retrieved_contexts=contexts)   # the reference is required
    recall = await ContextRecall(llm=llm).ascore(
        user_input=question, retrieved_contexts=contexts, reference=reference)   # the reference is required
    for name, r in (("faithfulness", faithfulness), ("answer_relevancy", relevancy),
                    ("context_precision", precision), ("context_recall", recall)):
        print(name, r.value)

asyncio.run(main())

Look at the arguments, not at the model names: Faithfulness does not accept reference, AnswerRelevancy does not accept the contexts, while ContextPrecision and ContextRecall require both. If your dataset has no reference, the last two cannot be computed and must not be replaced with an invented reference. Every call sends the judge model prompts that contain your questions and your documents: choose the model knowing where that data ends up, and compare costs on your real volume, because for a fixed number of calls per sample the cost grows linearly with the dataset.

How many examples you need and how wide the interval is

"I tested 100 questions and the faithfulness is 0.85" is a sentence with a missing part: the uncertainty. If the outcome of each question is yes or no (the right document is in the top three, the answer is supported), the score is a proportion, and for a proportion there are two formulas that are used as a pair. The first says how many questions you need to obtain a maximum error e with a given confidence: n = z² · p(1−p) / e², with p = 0.5 in the worst case because it maximises n. The second, the Wilson interval, says how wide the interval is that you actually obtained. In the listing z is computed from the standard library's normal distribution, with no copied tables.

# b2.py - how many questions are needed, and how wide is the interval of a score
import math
from statistics import NormalDist

def min_n(error: float, confidence: float = 0.95, p: float = 0.5) -> int:
    """n = z^2 * p * (1-p) / e^2, with p=0.5 as the worst case."""
    z = NormalDist().inv_cdf(1 - (1 - confidence) / 2)      # two-sided z
    return math.ceil(z * z * p * (1 - p) / (error * error))

def wilson(successes: int, n: int, confidence: float = 0.95) -> tuple[float, float]:
    """Wilson interval for a proportion: holds up better than 'p +- z*sqrt(p(1-p)/n)' near the edges."""
    z = NormalDist().inv_cdf(1 - (1 - confidence) / 2)
    p = successes / n
    centre = (p + z * z / (2 * n)) / (1 + z * z / n)
    half = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / (1 + z * z / n)
    return centre - half, centre + half

print("error   n@90%  n@95%  n@99%")
for e in (0.10, 0.07, 0.05, 0.03):
    print(f"±{e:.0%}   {min_n(e, .90):6d} {min_n(e, .95):6d} {min_n(e, .99):6d}")

for successes, n in ((17, 20), (24, 30), (85, 100), (170, 200), (340, 400)):
    lo, hi = wilson(successes, n)
    print(f"{successes}/{n} = {successes/n:.2f} -> 95% interval: [{lo:.2f}, {hi:.2f}]  (width {hi-lo:.2f})")

# Output:
# error   n@90%  n@95%  n@99%
# ±10%       68     97    166
# ±7%      139    196    339
# ±5%      271    385    664
# ±3%      752   1068   1844
# 17/20 = 0.85 -> 95% interval: [0.64, 0.95]  (width 0.31)
# 24/30 = 0.80 -> 95% interval: [0.63, 0.90]  (width 0.28)
# 85/100 = 0.85 -> 95% interval: [0.77, 0.91]  (width 0.14)
# 170/200 = 0.85 -> 95% interval: [0.79, 0.89]  (width 0.10)
# 340/400 = 0.85 -> 95% interval: [0.81, 0.88]  (width 0.07)

The first table confirms the order of magnitude of a rule of thumb: for ±7% at 95% you need 196 questions, for ±5% you need 385. Do not go below a hundred to reason about a comparison: with 100 questions 0.85 becomes an interval from 0.77 to 0.91, 14 points wide. With 20 questions the interval goes from 0.64 to 0.95, and with 30 questions at 0.80 (24 out of 30) from 0.63 to 0.90: too wide to tell apart two versions of the system that differ by five points. Doubling the questions from 100 to 200 narrows the interval from 14 to 10 points, it does not halve it: the width shrinks with the square root of n.

Two caveats that often go missing. The formulas hold for binary outcomes and independent questions: if the questions come from the same document in ten variants, the effective sample is smaller than the count suggests. And for a continuous score, such as an average faithfulness, the formula uses the standard deviation of the scores in place of p(1−p), which you must estimate from a first run. In any case decide the sample size before looking at the results; choosing the set of questions after seeing where the system does well is the quickest way to obtain a number that predicts nothing.

The judge is a model and has its flaws

Faithfulness and relevancy depend on a model that judges. Zheng and colleagues, in the paper that introduced MT-Bench and Chatbot Arena, studied exactly this use and described three distortions: the judge tends to prefer the answer presented first (position bias), the longer one (verbosity bias) and the one produced by models similar to itself (self-enhancement bias). The same authors found that a strong judge agrees with human preferences over 80% of the time, the same level of agreement as between people: the same study that denounces the flaws justifies the use, provided you know them.

Three habits follow. Do not change the judge between two runs that you want to compare, because a difference in score could be a difference in judge. Use a model different from the one that generates the answers when you can. Label a subset by hand and measure the agreement: with 30 cases, as the previous listing shows, the interval on the agreement is 28 points wide, so the check serves to discover a grossly wrong judge, not to certify a good one. The judge's absolute score travels poorly; the comparison between two versions of the same system, with the same judge and the same questions, travels far better.

What you can do now

You can tell the four failures of a RAG apart and associate with each the metric that sees it: dirty context with context precision, missing document with context recall, invention with faithfulness, off-topic answer with answer relevancy. You know that two of those metrics require a reference answer and two do not, and therefore where to start without a golden dataset. You can compute how many questions you need for the error you accept and how wide the interval you obtained is. And you have two zero-cost checks, on retrieval and on numbers, to put in your code this week. The next step is the tool that computes the metrics with a judge, in the article on Ragas, and then the way to bring them into a continuous-integration gate, in the one on DeepEval and Ragas; for production monitoring, which is not addressed here, there is the article Evaluating RAG: metrics, benchmarking and production monitoring, and for a compact version of a regression gate there is RAG Evaluation Pipeline: golden dataset.

The same series also includes production monitoring.