02 - Ragas framework: 4 metrics without golden label, synthetic data generation

A judge produces verdicts, the score is arithmetic

Ragas evaluates a RAG system without your having to write the right answer for every question. It is not magic: a model acts as the judge, breaks what it must evaluate into pieces and gives a verdict for each one, and the library reduces the verdicts to a number with simple arithmetic. Understanding this division changes the way you read the scores. The judge can be wrong, and you check that with a hand-labelled sample; the arithmetic is not wrong, and you can redo it with pen and paper. The first episode explained what the four metrics measure and what inputs they require; here we look inside, and answer three practical questions: where a NaN comes from, how to generate a test set without annotating by hand, and what an evaluation costs.

A note on method. No model was called to write this article. The formulas were rewritten in Python with NumPy and scikit-learn, with hand-written verdicts in place of the judge's, and then, for context precision and for faithfulness, compared with the internal calculation functions of version 0.4.3 of Ragas installed in a temporary folder: the values coincide (with a difference of the order of 10^-10 on the precision, due to a constant that protects against division by zero). Answer relevancy was not compared, because the library computes it with an embedding model. The library changes quickly, so always check the version you use: imports from ragas.metrics now emit a warning that points to ragas.metrics.collections.

The formulas redone by hand

The listing implements the four metrics without any model. context_precision receives a verdict for each chunk, in the order in which the retriever delivered them: 1 if the chunk helps to answer, 0 otherwise. It computes the precision at each position occupied by a useful chunk and averages it. Faithfulness and context recall share the same function because they have the same form, supported statements divided by total statements: the first counts the statements of the answer, the second those of the reference answer. answer_relevancy is the only one that works not with verdicts but with vectors: a few questions are generated starting from the answer and one measures how much they resemble the real question. Here the generated questions are written by hand and the vector is a TF-IDF, because an embedding would require a model; the absolute values, therefore, are not comparable with those of a real embedding.

# b1.py - the judge produces verdicts, the score is arithmetic: the four formulas with hand-written verdicts
import numpy as np
from sklearn.feature_extraction.text import TfidfVectorizer

def context_precision(verdicts: list) -> float:
    """Average precision at the relevant ranks. verdicts[i]=1 if the chunk at position i+1 is useful."""
    relevant = sum(verdicts)
    if relevant == 0:
        return 0.0                                           # no right chunk: a real zero, not a NaN
    return sum(sum(verdicts[: i + 1]) / (i + 1) for i, v in enumerate(verdicts) if v) / relevant

def statement_ratio(verdicts: list) -> float:
    """Faithfulness (statements of the answer) and context recall (statements of the reference).
    With no statements to count the result is NaN: the ratio is not defined, it is not zero."""
    return sum(verdicts) / len(verdicts) if verdicts else float("nan")

def answer_relevancy(question: str, generated_questions: list, noncommittal: list) -> float:
    """Mean of the cosines between the question and the questions generated from the answer; 0 if all are evasive."""
    tf = TfidfVectorizer().fit([question, *generated_questions])       # TF-IDF in place of an embedding
    v = tf.transform([question, *generated_questions]).toarray()
    cosines = v[1:] @ v[0] / (np.linalg.norm(v[1:], axis=1) * np.linalg.norm(v[0]))
    return float(cosines.mean()) * int(not all(noncommittal))

print("context_precision, right chunk at position 1:", round(context_precision([1, 0, 0]), 3))
print("context_precision, right chunk at position 2:", round(context_precision([0, 1, 0]), 3))
print("context_precision, right at positions 1 and 3:", round(context_precision([1, 0, 1]), 3))
print("context_precision, no right chunk:            ", context_precision([0, 0, 0]))
print("faithfulness, 2 of 3 statements supported:    ", round(statement_ratio([1, 1, 0]), 3))
print("faithfulness, no statement extracted:         ", statement_ratio([]))

Q = "How many users does the Pro plan include?"
on_topic = ["How many users does the Pro plan have?", "How many users are included in the Pro plan?", "Does the Pro plan include users?"]
generic = ["Which plans are available?", "Which plan should I choose?", "How does pricing work?"]
print("answer_relevancy, on-topic answer:   ", round(answer_relevancy(Q, on_topic, [False] * 3), 3))
print("answer_relevancy, generic answer:    ", round(answer_relevancy(Q, generic, [False] * 3), 3))
print("answer_relevancy, 'I don't know':    ", round(answer_relevancy(Q, on_topic, [True] * 3), 3))

# Output:
# context_precision, right chunk at position 1: 1.0
# context_precision, right chunk at position 2: 0.5
# context_precision, right at positions 1 and 3: 0.833
# context_precision, no right chunk:             0.0
# faithfulness, 2 of 3 statements supported:     0.667
# faithfulness, no statement extracted:          nan
# answer_relevancy, on-topic answer:    0.698
# answer_relevancy, generic answer:     0.132
# answer_relevancy, 'I don't know':     0.0

The first three lines show why context precision looks at the order. The same single useful chunk is worth 1.0 in first position and 0.5 in second; with two useful chunks, in first and third position, the average of the precisions at their ranks gives 0.833. If no chunk helps, the library returns 0.0 and not NaN: it is a real zero, the retriever failed. NaN is reserved for the cases where the ratio is not defined, as in faithfulness with no statements to count. For answer relevancy the on-topic answer gets 0.698 and the generic one 0.132, with this TF-IDF and these three invented examples; what matters is the distance between the two, not the decimals.

The last line has a consequence that surprises in practice. In the library's code the score is multiplied by zero when all the questions generated by the judge are classified as evasive (noncommittal): an honest answer such as "the context does not contain this information" receives answer relevancy 0. For the questions to which the system should answer "I don't know" this metric penalises the right behaviour. It is worth keeping these questions in a separate stratum of the dataset and evaluating them with another criterion, as the article on the golden dataset explains.

Where a NaN comes from

A NaN is not a low score: it is the absence of a score. Reading the source of version 0.4.3 and the documentation of evaluate(), the origins are few and recognisable.

• an exception during a call to the judge (invalid response format, timeout, provider error): with raise_exceptions=False the failed row becomes NaN and the job goes on;
• no statement extracted from the answer, for example an empty or one-word answer: faithfulness has nothing to count;
• no usable question generated from the answer, in the computation of answer relevancy;
• a null denominator in the cases of context recall, when the reference produces no statements.

The four causes call for different remedies. The first is a problem of the judge or of the infrastructure and is solved by looking at the logs, where the library records the exception of each job with its number; the other three are data problems, and must be solved upstream with valid samples. A clarification that contradicts a common piece of advice: raise_exceptions is already False by default in evaluate(), so it is not an option to switch on in order to "not get stuck"; on the contrary, in the test set generation method the default value is True, and there it is worth deciding it consciously.

The point that weighs most remains: what you do with the NaNs when you report a result. Excluding them silently amounts to assuming that the failed samples would have behaved like the others, and usually that is not true, because a judge fails more often on long contexts and on hard questions. The listing shows the effect with ten example scores, three of which are NaN.

# b2.py - NaNs are not zeros and they are not absent: report them, and give the interval where the mean can lie
import math

# faithfulness of 10 samples; NaN = the judge failed or extracted no statements
scores = [1.0, 0.9, 1.0, 0.8, math.nan, 1.0, math.nan, 0.9, math.nan, 1.0]

valid = [p for p in scores if not math.isnan(p)]
n_nan = len(scores) - len(valid)

mean_on_valid = sum(valid) / len(valid)           # optimistic assumption: the failed ones would have done like the others
mean_nan_as_zero = sum(valid) / len(scores)       # pessimistic assumption: all the failed ones were terrible answers

print(f"samples={len(scores)}  NaN={n_nan} ({n_nan / len(scores):.0%})  valid={len(valid)}")
print(f"mean on valid:      {mean_on_valid:.3f}")
print(f"mean with NaN as 0: {mean_nan_as_zero:.3f}")
print(f"the true mean lies between {mean_nan_as_zero:.2f} and {mean_on_valid:.2f}")

# Output:
# samples=10  NaN=3 (30%)  valid=7
# mean on valid:      0.943
# mean with NaN as 0: 0.660
# the true mean lies between 0.66 and 0.94

The mean on the valid samples alone is 0.943, the one that counts the NaNs as zeros is 0.660: the true mean, which you do not know, lies in between, and the width of the interval (28 points) far exceeds any improvement you might want to demonstrate. The rule that follows is to always report the share of NaN next to the mean and, if it is high, to solve the why before comparing versions of the system. There is no acceptable NaN threshold valid for everyone: the threshold is the share beyond which the interval becomes wider than the difference you are looking for.

Synthetic data: useful for getting started, with a construction flaw

TestsetGenerator starts from your documents, builds a knowledge graph from them and asks a model to write questions and reference answers. The documentation gives as the default distribution three types of question: single-hop on a specific detail (half of the total), multi-hop on an abstract topic (a quarter) and multi-hop on specific details (a quarter). The old "simple, reasoning, multi-context" taxonomy of the early versions no longer exists; if you find it in a tutorial, the tutorial is dated. The listing is a schema to adapt and was not run, because it requires a model and your documents.

# b3.py - schema to adapt (not run): you need a judge model, an embedding model and your own documents
# Signatures checked with inspect.signature on ragas 0.4.3; no call to a model was made.
from langchain_community.document_loaders import DirectoryLoader
from ragas import EvaluationDataset, evaluate
from ragas.embeddings import LangchainEmbeddingsWrapper
from ragas.llms import LangchainLLMWrapper
from ragas.metrics import Faithfulness, LLMContextPrecisionWithReference, LLMContextRecall, ResponseRelevancy
from ragas.testset import TestsetGenerator
from ragas.testset.synthesizers import default_query_distribution

generator_llm = LangchainLLMWrapper(LANGCHAIN_CHAT_MODEL)             # your model, via LangChain
generator_emb = LangchainEmbeddingsWrapper(LANGCHAIN_EMBEDDINGS)

docs = DirectoryLoader("documents/", glob="**/*.md").load()
generator = TestsetGenerator(llm=generator_llm, embedding_model=generator_emb)
testset = generator.generate_with_langchain_docs(
    docs,
    testset_size=30,
    query_distribution=default_query_distribution(generator_llm),   # single-hop, specific and abstract multi-hop
    raise_exceptions=False,                                          # the default of this method is True
)
rows = testset.to_pandas()          # user_input, reference_contexts, reference, synthesizer_name

# Now have YOUR system answer and complete each row with the answer and the retrieved contexts
samples = []
for r in rows.to_dict("records"):
    answer, contexts = YOUR_RAG(r["user_input"])
    samples.append({"user_input": r["user_input"], "reference": r["reference"],
                    "response": answer, "retrieved_contexts": contexts})

result = evaluate(
    dataset=EvaluationDataset.from_list(samples),
    metrics=[LLMContextPrecisionWithReference(), LLMContextRecall(), Faithfulness(), ResponseRelevancy()],
    llm=generator_llm, embeddings=generator_emb,
    raise_exceptions=False,              # already the default of evaluate(): failures become NaN
    token_usage_parser=TOKEN_PARSER_OF_YOUR_PROVIDER,                # without it, total_tokens() raises an error
)
print(result.to_pandas().isna().mean())    # share of NaN per metric
print(result.total_tokens())               # tokens actually consumed by the judge

Three lines deserve attention. query_distribution decides the proportion of the question types; if you leave the default value, you accept the split quoted above. raise_exceptions=False in generation prevents a badly produced question from stopping the whole batch. The last pair of lines is the one that most examples omit: token_usage_parser is what allows total_tokens() to say how much the judge consumed. Without it, the call raises an error, as the library's code indicates.

The construction flaw is this: every synthetic question is born from a document that contains it, so it always has an answer in the corpus. Missing are the questions without an answer, those phrased with the users' words and those about things the corpus does not cover, which are precisely the cases in which a RAG fails. Moreover the reference answer is written by a model, and if it is the same one that then acts as judge you risk the self-preference bias described in the paper by Zheng. Read in full the first batch of thirty questions, discard the ambiguous ones and correct the wrong references: the synthetic dataset is a draft that speeds up the work, not a golden dataset.

What an evaluation costs

The cost is not estimated in advance with a round figure: it is measured on a small sample and multiplied. The reason lies in the number of calls each metric makes to the judge. In the code, context precision with a reference queries the judge once for each retrieved chunk; faithfulness makes at least two calls per sample, one to extract the statements and one to judge them; answer relevancy generates several questions per sample (three by default, parameter strictness) and asks for the embeddings. It follows that the cost grows with the number of chunks and with the length of the answers, not only with the number of samples: doubling the top-k doubles the calls of the precision.

Run first on ten samples with token_usage_parser active, read total_tokens(), multiply by the number of samples and by the price of your judge model. Time is measured in the same way. With a local model the monetary cost drops to zero but the time grows, and small models produce more invalid responses, therefore more NaN: cost, time and share of NaN are paid together, and the cheapest judge is not always the one that costs least in total.

What you can do now

You can redo by hand the four Ragas metrics starting from the judge's verdicts and you know why context precision depends on the order, while answer relevancy penalises an honest "I don't know". You can recognise the four origins of a NaN, report it next to the mean with the interval that follows from it, and tell a default of evaluate() from one of generate_with_langchain_docs(). You can generate a synthetic test set and you know its structural limit. The next step is to bring these metrics into your continuous-integration pipeline, with DeepEval compared with Ragas; for the metrics of the retriever alone, there is the article on NDCG, MRR and HitRate.

The same series also includes the faithfulness metric.