RAG evaluation on-prem: pipeline and metrics RAGAS DeepEval

How do you know your RAG works when nobody is watching it

An internal assistant that answers with confidence can be wrong in two very different ways: the search system brings the wrong documents to the model, or the right documents arrive but the model adds something of its own. If you only look at the final answer you do not know which of the two faults you have, and so you do not know what to repair. Evaluating a RAG means measuring the two pieces separately, on a set of questions whose right answer you know, and doing it again every time you change something.

This article builds that method for an SME that keeps its documents in-house. First the four metrics as the RAGAS documentation defines them, with a program that computes them by hand to show you where the catch is. Then the golden dataset, that is the questions with the reference answer: how many you really need is calculated, not guessed. Then the judge, and this is where the common idea "RAGAS does not need an LLM" turns out to be false: all the metrics that follow use a model as judge, and where that model runs is a decision that concerns your data. Finally the comparison between two configurations and monitoring over time.

If you are missing the upstream piece, that is how the index and the document retrieval are built, start from Complete setup of Ollama + RAG + RediSearch. For the choice of the model that will act as generator and judge, the comparison between Llama, Mistral, Qwen and Gemma is in the previous article of the series.

The four metrics as RAGAS defines them

RAGAS is an open source library for evaluating RAG pipelines. Its four most used metrics split into two pairs: two look at retrieval (how good the set of text fragments, the chunks, that reaches the model is) and two look at generation (how good the answer is). The table says what each one needs and, a point that matters for your infrastructure, what a model contributes to it.

The table summarises the metric pages of the RAGAS documentation, consulted on 2026-11-13 (the sources are at the bottom). The useful reading is this: two metrics out of four (precision and recall) require a reference answer, and all four go through an LLM. So "evaluating on-prem" means running the judge in-house too, not just the system being evaluated.

Faithfulness: does the answer say only things that the documents support?

The documentation defines the score as the number of claims in the answer supported by the context divided by the total number of claims. The official example is Einstein's date of birth: the answer "born in Germany on 20 March 1879" contains two claims, the first is supported, the second is not because the context says 14 March, and the score is 1/2 = 0.5. Note that faithfulness does not measure whether the answer is true in the world, only whether the documents the system found support it: a correct answer that cannot be derived from the chunks scores zero.

Context precision: the order counts, not only the set

The version that circulates most often (relevant found divided by total found) is not the RAGAS one. The documentation defines it as the mean of the precisions computed at each position where a relevant chunk appears, so a useful chunk in first place is worth much more than the same chunk in fifth. It is the behaviour you really need, because the model reads the first chunks with more attention and the context has a length limit.

Context recall: did you lose something important?

Here too the definition is by claims: the reference answer is broken down and you count how many claims are found in the retrieved chunks. If the reference says three things and the chunks support two, the retrieval scores 2/3. This is why the golden dataset must contain a well-written reference answer, and not only the list of expected documents.

Answer relevancy: does the answer talk about what you asked?

Starting from the answer, the judge generates some artificial questions (three by default) and RAGAS computes the cosine similarity between each of them and the real question, then takes the mean. An evasive answer, or one full of digressions, produces questions far from the original and therefore a low score. It does not measure correctness: a false but pertinent answer gets a high score.

The formulas, run on judgements already given

To understand where the catches are, it is worth computing the metrics without any model, starting from judgements already given: every claim and every chunk is a boolean that in production would come from the judge. The program below contains three functions, one per metric. In context_precision the line that matters is the one that adds veri_positivi / k only when the chunk at position k is relevant: remove it and you are back to the naive formula that ignores order. The last experiment uses the same five chunks, with only two relevant, but in a different order.

"""The four RAG metrics computed by hand on judgements already given."""


def faithfulness(giudizi):
    """giudizi: one boolean per claim in the answer (True = supported by the context)."""
    return sum(giudizi) / len(giudizi)


def context_recall(giudizi_riferimento):
    """One boolean per claim in the reference answer (True = found in the chunks)."""
    return sum(giudizi_riferimento) / len(giudizi_riferimento)


def context_precision(rilevanti):
    """rilevanti: one boolean per chunk, in rank order. Weighted mean of precision@k."""
    veri_positivi = 0
    somma = 0.0
    for k, ok in enumerate(rilevanti, start=1):
        if ok:
            veri_positivi += 1
            somma += veri_positivi / k
    return somma / veri_positivi if veri_positivi else 0.0


# Einstein example: two claims, one with the wrong date
print("faithfulness  :", faithfulness([True, False]))

# Same 5 chunks, only two relevant, different order
print("precision A   :", round(context_precision([True, True, False, False, False]), 3))
print("precision B   :", round(context_precision([False, False, False, True, True]), 3))

# Reference answer with 3 claims, 2 found in the chunks
print("recall        :", round(context_recall([True, True, False]), 3))

# Output:
# faithfulness  : 0.5
# precision A   : 1.0
# precision B   : 0.325
# recall        : 0.667

The output reads like this. The faithfulness of 0.5 reproduces the Einstein case. The two precisions are the surprise: same chunks, same number of relevant ones, but 1.0 if the two useful ones are at the top and 0.325 if they are at the bottom, because positions 4 and 5 give partial precisions of 1/4 and 2/5. This is why a change of reranker, which reorders without adding or removing documents, shows up in context precision and not in recall. The code does not depend on any model and gives the same output on every run (verified with two consecutive launches). The real judge replaces only the booleans.

The golden dataset: how many questions you need and how to check them

The golden dataset is the set of questions for which someone has already decided the right answer and which documents contain it. Each entry has four fields: the question, the reference answer, the identifiers of the expected documents, and a category label that then lets you read the scores by area. The questions come from real tickets and chats, because the ones invented at a desk resemble the questions you expect, not the ones that arrive.

What remains is the question of how many entries are needed. The honest answer is that it depends on how much uncertainty you are willing to tolerate on the score. With few questions a score of 80% can hide a true value far from it. The program below computes the 95% Wilson confidence interval (the same one that scipy.stats.binomtest(...).proportion_ci(method='wilson') returns: I verified that for 80 out of 100 both give 0.711 to 0.867) and, in the first part, checks a small golden dataset built for the example. The three defects inserted are the ones you really find: a duplicated question, an empty reference, an expected document that no longer exists in the knowledge base.

"""Golden dataset check and width of the uncertainty as a function of size."""
import math

# Constructed example: three entries of a company's internal assistant (data invented for the example)
KB = {"kb-101", "kb-102", "kb-205"}
GOLDEN = [
    {"id": "g1", "query": "How do I reset my password?",
     "reference": "From the login form, the Forgot password link.", "expected_ids": ["kb-101"]},
    {"id": "g2", "query": "How do I reset my password?",
     "reference": "From the login form.", "expected_ids": ["kb-101"]},
    {"id": "g3", "query": "Where do I find the March invoice?",
     "reference": "", "expected_ids": ["kb-999"]},
]


def controlla(golden, kb):
    problemi, viste = [], {}
    for voce in golden:
        if not voce["reference"].strip():
            problemi.append(f"{voce['id']}: empty reference answer")
        for doc in voce["expected_ids"]:
            if doc not in kb:
                problemi.append(f"{voce['id']}: expected document {doc} does not exist in the knowledge base")
        q = voce["query"].strip().lower()
        if q in viste:
            problemi.append(f"{voce['id']}: same question as {viste[q]}")
        viste.setdefault(q, voce["id"])
    return problemi


def wilson(successi, n, z=1.96):
    p = successi / n
    centro = (p + z * z / (2 * n)) / (1 + z * z / n)
    meta = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / (1 + z * z / n)
    return centro - meta, centro + meta


for p in controlla(GOLDEN, KB):
    print("PROBLEM", p)

print("--- 95% interval if the system answers 80% of the questions well")
for n in (30, 100, 300, 1000):
    lo, hi = wilson(round(0.8 * n), n)
    print(f"n={n:4d}  from {lo:.3f} to {hi:.3f}  (width {hi - lo:.3f})")

print("--- annotation time, assumption: 10 minutes per entry")
for n in (30, 100, 300):
    print(f"n={n:3d} -> {n * 10 / 60:.1f} hours")

# Output:
# PROBLEM g2: same question as g1
# PROBLEM g3: empty reference answer
# PROBLEM g3: expected document kb-999 does not exist in the knowledge base
# --- 95% interval if the system answers 80% of the questions well
# n=  30  from 0.627 to 0.905  (width 0.278)
# n= 100  from 0.711 to 0.867  (width 0.155)
# n= 300  from 0.751 to 0.841  (width 0.090)
# n=1000  from 0.774 to 0.824  (width 0.050)
# --- annotation time, assumption: 10 minutes per entry
# n= 30 -> 5.0 hours
# n=100 -> 16.7 hours
# n=300 -> 50.0 hours

The second half of the output is the figure you need in order to decide. With 100 entries the interval is 15.5 percentage points wide (from 71% to 87%): a system that scores 80% could be worth 72 or 86. With 300 entries it drops to 9 points, with 1,000 to 5. So "100 questions are enough" is true only if you are content to tell a good system from a terrible one, not two close configurations. The cost of annotation is linear: with the assumption, yours to replace with a measurement on your own team, of 10 minutes per entry, 100 entries are 16.7 hours of work by a person competent in the domain. Delegating annotation to strangers makes sense for generic content, not for a tax or legal domain, where the right answer is known by whoever works there.

The judge in-house: RAGAS and DeepEval with a local model

Since every metric goes through an LLM, the company's documents and the generated answers end up in the judge's prompt. If the judge is an external service, every evaluation sends your content outside your infrastructure, with all the data protection consequences that the article on privacy by design lines up. A local judge avoids the problem but has a different cost: a small model judges worse than a large one, and RAGAS and DeepEval ask it for structured JSON that small models get wrong more often.

RAGAS with Ollama

The RAGAS guide to LLM adapters shows exactly this case: Ollama exposes an API compatible with OpenAI's, so you use the OpenAI client with a local base_url and the llm_factory function with provider="openai". The key is mandatory for the client but Ollama ignores it, which is why any string will do. The schema below is taken from the documentation and adapted to faithfulness; I did not run it because it would need an Ollama server and a downloaded model, and for that reason I report no score.

# Schema to adapt, NOT run: requires ragas, an Ollama server and a downloaded model.
from openai import OpenAI
from ragas.llms import llm_factory
from ragas.metrics.collections import Faithfulness

# Ollama exposes an OpenAI-compatible API; the key is required by the client but ignored.
client = OpenAI(api_key="ollama", base_url="http://localhost:11434/v1")
llm = llm_factory("mistral", provider="openai", client=client)

scorer = Faithfulness(llm=llm)
risultato = scorer.score(
    user_input="Where was Einstein born and when?",
    response="Einstein was born in Germany on 20 March 1879.",
    retrieved_contexts=["Albert Einstein (born 14 March 1879) was a German theoretical physicist."],
)
print(risultato.value)  # the score depends on the judge model: we do not report a value

Two warnings on the code. RAGAS versions change API often: the documentation itself marks the old path from ragas.metrics import ... as legacy and recommends ragas.metrics.collections, so check the page for the version you install. And the model you pass to llm_factory must exist on your server: "mistral" is the name the documentation example uses, not a recommendation.

DeepEval with a custom judge

DeepEval has its own metrics, with different names (FaithfulnessMetric, AnswerRelevancyMetric) and the same principle: an LLM judges. By default it uses an OpenAI model (the page consulted on 2026-11-13 indicates gpt-5.4 as the default value of model), but it accepts any class that inherits from DeepEvalBaseLLM. The guide to custom models lists the rules: implement get_model_name, load_model, generate and a_generate, and accept a schema parameter, because without it the metrics that expect a structured object fail with an unclear error. The branch with the schema uses Ollama's format field, which accepts a JSON schema and constrains the model's output.

# Schema to adapt, NOT run: requires deepeval, ollama (Python client) and a local model.
from pydantic import BaseModel
from ollama import Client
from deepeval.models import DeepEvalBaseLLM
from deepeval.metrics import FaithfulnessMetric
from deepeval.test_case import LLMTestCase


class GiudiceLocale(DeepEvalBaseLLM):
    def __init__(self, modello: str):
        self.modello = modello
        self.client = Client(host="http://localhost:11434")

    def load_model(self):
        return self.client

    def generate(self, prompt: str, schema: BaseModel | None = None):
        messaggi = [{"role": "user", "content": prompt}]
        if schema is None:
            return self.client.chat(model=self.modello, messages=messaggi).message.content
        # With the schema Ollama constrains the output to valid JSON
        r = self.client.chat(model=self.modello, messages=messaggi, format=schema.model_json_schema())
        return schema.model_validate_json(r.message.content)

    async def a_generate(self, prompt: str, schema: BaseModel | None = None):
        return self.generate(prompt, schema)

    def get_model_name(self):
        return self.modello


metrica = FaithfulnessMetric(threshold=0.7, model=GiudiceLocale("mistral"), include_reason=True)
caso = LLMTestCase(
    input="Can I return the shoes?",
    actual_output="Yes, within 30 days and at no extra cost.",
    retrieval_context=["All customers are entitled to a refund within 30 days, at no extra cost."],
)
metrica.measure(caso)
print(metrica.score, metrica.reason)

This too is a schema to adapt, not run. The most important line is the one in generate that passes schema.model_json_schema() to format: without the constraint a small model answers with free text around the JSON and DeepEval cannot read it. When your judge gives strange results, the first suspect is always the output format.

RAGAS or DeepEval, and G-Eval

Neither of the two is "more accurate" in absolute terms: the percentage accuracy figures that circulate for the two frameworks do not come from a source that I could verify, and I do not report them. The verifiable differences are others.

• RAGAS offers the retrieval and generation metrics described above, with documented formulas, and is designed for the "dataset in, scores out" flow.
• DeepEval has a test-style approach: every case has a threshold (0.5 by default), a pass or fail result, and a motivation in text; it integrates into a test suite in CI.
• DeepEval also includes G-Eval, a metric in which you describe the criterion in natural language and the judge produces the score.

As a selection criterion: if you want a dashboard of retrieval and generation with standard metrics, RAGAS; if you want quality to block a change the way a test would, DeepEval. You can use both on the same data. G-Eval comes from the paper by Liu and others (2023), which proposes an LLM judge with chain of reasoning and form filling; the DeepEval documentation presents it as the way to evaluate subjective criteria specific to your case, to be placed alongside the system metrics. The corollary for regulated domains: a local judge is not a lawyer or a doctor. A criterion such as "is it legally correct" can produce a score, but the score must be compared with the judgement of a competent person before you trust it.

How far to trust the local judge

Before relying on the scores, measure how much your judge agrees with a person. Take 30-50 entries of the golden dataset, have the faithfulness of each answer judged by hand, compare with the judge's verdict and compute the percentage of agreement. If the judge contradicts the human more than once in five, change model or simplify the prompt before building automatic alarms on top of it. It is a check you do once, and it saves you from months of charts that look precise and are not.

Comparing two configurations without fooling yourself

Scenario: the system uses 512-token chunks and you want to try shorter chunks. The naive approach splits the questions at random into two groups, one configuration per group, and compares the means. The flaw is that questions have different difficulties: if by chance the hardest ones end up in one group, you see a difference that has nothing to do with the chunks. The solution is to run both configurations on the same questions and compare, question by question, who answered well.

The test suited to this paired scheme is McNemar's, which looks only at the questions on which the two configurations differ (the discordant ones): if they are equal in number in favour of one and of the other, there is no difference. In SciPy the exact form is obtained with binomtest on the discordant pairs; the mcnemar function is not in scipy.stats but in StatsModels. The program's numbers are constructed for the example: 100 questions, 62 right in both, 19 wrong in both, 14 right only with the short chunks and 5 only with the long ones.

"""Comparison of two RAG configurations on the SAME 100 questions (exact McNemar test)."""
from scipy.stats import binomtest

# Outcome per question: 1 = answer judged correct. Data constructed for the example:
# 62 questions correct in both, 19 wrong in both, 5 only with chunk 512, 14 only with chunk 256.
solo_512 = 5
solo_256 = 14
entrambe_ok = 62
entrambe_ko = 19
n = solo_512 + solo_256 + entrambe_ok + entrambe_ko

print(f"questions: {n}")
print(f"correct with 256: {entrambe_ok + solo_256}   with 512: {entrambe_ok + solo_512}")

# Only the questions on which the two configurations DIFFER count (the discordant ones).
discordanti = solo_512 + solo_256
esito = binomtest(solo_256, discordanti, p=0.5, alternative="two-sided")
print(f"discordant: {discordanti}, won by 256: {solo_256}")
print(f"exact p-value: {esito.pvalue:.4f}")

# Same difference, but with only 4 vs 9 discordant: the verdict changes.
esito_piccolo = binomtest(9, 13, p=0.5)
print(f"with 9 vs 4 discordant: p-value {esito_piccolo.pvalue:.4f}")

# Output:
# questions: 100
# correct with 256: 76   with 512: 67
# discordant: 19, won by 256: 14
# exact p-value: 0.0636
# with 9 vs 4 discordant: p-value 0.2668

The result is counterintuitive and is the reason the test is done: 76 right answers against 67 is a difference of nine points, but the exact p-value is 0.0636, above the usual 5% threshold. With that sample you cannot say that the short chunks win. The last line shows the other face: with 9 discordant pairs in favour against 4 the p-value is 0.27, confirming that what counts is the discordant pairs, not the gross difference. The practical consequence is to enlarge the golden dataset before deciding, not to pick the configuration that "seems" better. Remember too that a threshold of 0.05 is a convention: declare it before seeing the data.

Watching for drift over time

A RAG degrades even when nobody touches the code: documents change, the golden dataset ages, an update replaces the embedding model. The idea is to run the evaluation on the golden dataset every night (one cron line, below) and record the scores, then see whether the last few days deviate from the reference baseline. A fixed threshold such as "alarm below 0.75" is fragile: it depends on the normal level of your system, which can be 0.60 or 0.92. It is better to compare with the mean and the dispersion of the preceding days.

# Schema to adapt, NOT run. Every night at 02:00 (server time) evaluate the golden dataset
# and append a row to the scores file; the ragas_eval.py script is the one in your pipeline.
0 2 * * * /opt/rag/venv/bin/python /opt/rag/ragas_eval.py --golden /opt/rag/golden.json --append /var/lib/rag/punteggi.jsonl

The line is a schema to adapt and I did not run it: the five fields mean minute, hour, day of the month, month and day of the week, so 0 2 * * * is "at 02:00 every day". The script and the paths are those of your pipeline; leave the scores in a JSON-lines file, which can be appended to without rewriting. The real detector, on the other hand, is pure logic and can be run.

The program uses 14 days as the baseline and the three most recent measurements as the window, and raises the alarm when the recent mean drops below the baseline mean minus three standard deviations. The series is constructed for the example: fifteen stable days around 0.85 and then a drop to 0.71, 0.69, 0.72. The second line of output verifies that, without the drop, the detector does not fire: an alarm that always sounds gets ignored.

from statistics import mean, stdev

# Daily context recall scores over a week and a half: SERIES CONSTRUCTED for the example.
punteggi = [0.84, 0.86, 0.85, 0.83, 0.87, 0.85, 0.84, 0.86, 0.85, 0.84,
            0.85, 0.86, 0.84, 0.85, 0.83, 0.71, 0.69, 0.72]


def deriva(serie, base_giorni=14, finestra=3, k=3.0):
    base = serie[:base_giorni]
    recenti = serie[-finestra:]
    media_base, dev_base = mean(base), stdev(base)
    soglia = media_base - k * dev_base
    return {
        "media_base": round(media_base, 3),
        "dev_base": round(dev_base, 3),
        "soglia": round(soglia, 3),
        "media_recente": round(mean(recenti), 3),
        "allarme": mean(recenti) < soglia,
    }


print(deriva(punteggi))
print("without the drop:", deriva(punteggi[:15], finestra=1)["allarme"])

# Output:
# {'media_base': 0.849, 'dev_base': 0.011, 'soglia': 0.817, 'media_recente': 0.707, 'allarme': True}
# without the drop: False

With baseline 0.849, deviation 0.011 and threshold 0.817 the recent mean of 0.707 triggers the alarm. The k parameter sets how sensitive the detector is: with a lower k it fires sooner but is wrong more often. Two limits to know: if the baseline already contains a degraded period, the threshold drops along with it, and with a three-day window a single anomalous score can be enough, so it is worth requiring that the drop last more than one day. An alarm says that something has changed, not what.

When the score collapses: where to look first

Suppose context recall goes from 0.85 to 0.62 from one day to the next. The order of checks is best both from the cheapest to the most expensive and from what most probably changed. One reading rule underpins the whole scheme: if the retrieval metrics drop, the fault lies upstream of the model, and changing the generator does not help.

1. Are the golden dataset and the document base still aligned? Documents that were moved, renamed or deleted make the expected references disappear, and recall collapses without the system having got worse.
2. Is the embedding model the same one the index was built with? Changing it without reindexing compares vectors from different spaces.
3. Is the index intact? With RediSearch the FT.INFO command returns information and statistics on the index, including the number of documents, to be compared with the expected one.
4. Have the retrieval parameters changed, for example the number of chunks returned?
5. Has the judge changed? An update of the judge model shifts all the scores without the evaluated system having changed.

The last item is the most insidious: that is why it is worth recording, next to every score, the name and version of the judge model and of the embedding model. When two consecutive points of the series have different judges, the jump must not be compared. For repairing the index do not type FLUSHDB with your eyes closed: it deletes all the keys of the selected database, not only the index, and sessions and caches are in there too. Rebuild the index with the dedicated commands and verify with FT.INFO when the rebuild is finished.

Frequently asked questions

Does RAGAS need an LLM?

Yes. All four metrics described here call a judge LLM, to break down the claims or to judge the chunks, and faithfulness and context recall cannot be computed without one. If you want to keep the data in-house, the judge must be a local model.

How many questions are needed in the golden dataset?

It depends on the precision you need. With 100 entries a score of 80% has a 95% interval from 71% to 87%, with 300 from 75% to 84%. A few dozen are enough to notice a big fault, not to compare two close configurations.

Can I evaluate with the same model that generates the answers?

You can, but a model tends to judge its own style leniently. If you can, use a different model as judge and measure the agreement with human labels on a sample before trusting the scores.

What you can do now

You can tell a retrieval fault from a generation fault, because the four metrics separate them and you know how they are really computed. You can build a golden dataset and say how credible a score is as a function of how many entries it contains. You can connect RAGAS and DeepEval to a local judge without sending the documents outside, and you know that a small judge has to be calibrated on human labels. You can compare two configurations with a paired test and avoid deciding on a difference that is noise, and you can monitor the system over time with a reference baseline instead of a fixed threshold. The next step in the series is load, in from 1 to 50 concurrent users, and then what to do when the system stops, in disaster recovery.

The same series also includes the cases where a managed service is the better choice and the recap with the decision sheet.