03 - DeepEval vs Ragas: when to choose what, CI/CD integration with pytest

You do not choose a winner, you choose where to put the gate

Ragas and DeepEval do largely overlapping things: both compute faithfulness, answer relevancy and context precision and recall with a model as the judge. The difference that affects everyday work lies not in the metrics but in the point where they plug in. Ragas was born to evaluate a batch of samples and return a table of scores; DeepEval presents itself as a test: an assert_test function that fails with a readable reason and a command, deepeval test run, that runs pytest and returns its exit code. If your problem is "knowing how it is going", the first is enough. If it is "preventing a change from making the system worse", you need the second, or glue written by you around the first.

This article starts from the case of Ragas and brings the metrics into a continuous-integration gate. The method is the reverse of the usual one: the first listing uses no model. It builds a gate with two deterministic metrics written by you and actually runs it, because the part that breaks first in CI is not the judge, it is the scaffolding: thresholds, exit codes, error messages. The model-based judge comes afterwards, as a schema to adapt. The DeepEval signatures were read on version 4.3.4 installed in a temporary folder; the listing with the models was not run because no model is called.

What they have in common and where they differ

The table contains only what I could verify in the documentation and in the code of the two libraries. It does not compare speed, prices or quality of judgement: I did not measure them.

The row that weighs most is the threshold one. Ragas returns numbers and the decision stays with you; DeepEval embeds the decision (threshold) inside the metric, and that is why it lends itself to a test that passes or fails. Two caveats that the table cannot contain. First: metrics with the same name are not interchangeable, because the prompts and the internal steps are different, so a faithfulness of 0.82 in one library is not worth a 0.82 in the other; do not mix the historical series. Second: if you do not pass a model, the DeepEval documentation gives an OpenAI model as the default, which means a key from that vendor and sending your documents to that vendor. Always pass the judge explicitly, and choose it knowing where the data goes.

A gate without a model: two metrics, four cases

The listing defines two metrics by extending BaseMetric, DeepEval's base class. A custom metric must provide measure, the asynchronous version a_measure, is_successful and the name. RetrievalMetric is worth 1 if at least one expected document appears among the first k retrieved; the identifiers travel in the metadata field of the test case. NumbersMetric is the check on numbers from episode 1, here expressed as the share of numbers in the answer that appear in the context. Both have threshold=1.0: a case passes only if all the numbers are supported and if retrieval succeeded.

# b1.py - a quality gate with assert_test and two deterministic metrics (no model)
import os, re
os.environ["DEEPEVAL_TELEMETRY_OPT_OUT"] = "YES"      # no telemetry from the test machine
from deepeval import assert_test
from deepeval.metrics import BaseMetric
from deepeval.test_case import LLMTestCase

class RetrievalMetric(BaseMetric):
    """1 if an expected document is among the top k retrieved, otherwise 0. The ids are in metadata."""
    def __init__(self, k: int = 3, threshold: float = 1.0):
        self.k, self.threshold = k, threshold
    def measure(self, test_case: LLMTestCase) -> float:
        expected = set(test_case.metadata["expected_ids"])
        found = set(test_case.metadata["retrieved_ids"][: self.k])
        self.score = 1.0 if expected & found else 0.0
        self.success = self.score >= self.threshold
        self.reason = f"expected={sorted(expected)} retrieved_top{self.k}={test_case.metadata['retrieved_ids'][: self.k]}"
        return self.score
    async def a_measure(self, test_case: LLMTestCase) -> float:
        return self.measure(test_case)
    def is_successful(self) -> bool:
        return self.success
    @property
    def __name__(self):
        return "Retrieval"

class NumbersMetric(BaseMetric):
    """Score = share of the numbers in the answer that appear in the context (1.0 if the answer has none)."""
    def __init__(self, threshold: float = 1.0):
        self.threshold = threshold
    def measure(self, test_case: LLMTestCase) -> float:
        context = " ".join(test_case.retrieval_context)
        in_answer = re.findall(r"\d+(?:[.,]\d+)?", test_case.actual_output)
        ok = [n for n in in_answer if n in re.findall(r"\d+(?:[.,]\d+)?", context)]
        self.score = len(ok) / len(in_answer) if in_answer else 1.0
        self.success = self.score >= self.threshold
        self.reason = f"numbers in the answer={in_answer}, supported={ok}"
        return self.score
    async def a_measure(self, test_case: LLMTestCase) -> float:
        return self.measure(test_case)
    def is_successful(self) -> bool:
        return self.success
    @property
    def __name__(self):
        return "Numbers"

def case(i, question, answer, context, expected, retrieved):
    return LLMTestCase(input=question, actual_output=answer, retrieval_context=context,
                       metadata={"expected_ids": expected, "retrieved_ids": retrieved}, name=i)

CASES = [
    case("q1", "How much does the Pro plan cost?", "It costs 49 euros a month.", ["The Pro plan costs 49 euros a month."], ["pricing"], ["pricing", "api"]),
    case("q2", "How many users does Pro include?", "It includes up to 25 users.", ["The Pro plan costs 49 euros a month."], ["users"], ["pricing", "api"]),
    case("q3", "How many requests per minute?", "The limit is 100 requests per minute.", ["The API accepts 100 requests per minute."], ["api"], ["sso", "api"]),
    case("q4", "How do I reset my password?", "Use the link sent by email.", ["To reset your password use the link sent by email."], ["password"], ["invoices", "api", "sso", "password"]),
]

def test_case(c: LLMTestCase):
    """With pytest this would be @pytest.mark.parametrize("c", CASES) and you would launch 'deepeval test run file.py'."""
    assert_test(c, [RetrievalMetric(k=3), NumbersMetric()], run_async=False)

failures = 0
for c in CASES:
    try:
        test_case(c)
        print(f"{c.name}: PASSES")
    except AssertionError as e:
        failures += 1
        failed = re.findall(r"(\w+) \(score: ([\d.]+)", str(e))      # names and scores of the metrics below threshold
        print(f"{c.name}: FAILS -> " + ", ".join(f"{n}={s}" for n, s in failed))

rate = 1 - failures / len(CASES)
print(f"success rate={rate:.0%}  gate threshold=75%  ->  {'merge allowed' if rate >= 0.75 else 'merge blocked'}")

# Output:
# q1: PASSES
# q2: FAILS -> Retrieval=0.0, Numbers=0.0
# q3: PASSES
# q4: FAILS -> Retrieval=0.0
# success rate=50%  gate threshold=75%  ->  merge blocked

The gate is in the last line, not in assert_test. Each single case passes or fails, and the error message lists the metrics below threshold with their score: q2 fails on both (the retriever did not bring the document on users and the answer invents "25 users"), q4 fails only on retrieval. The useful detail is q4: the answer is right, but the expected document was in fourth place, outside k=3. The case is not a generator error, it is a retriever that delivers the result too late, and the test says so with the name of the metric. Then the decision is made at the level of the whole set: the 50% success rate is below the 75% threshold and the merge is blocked. A threshold per case would be fragile with a model-based judge, which for the same input can give slightly different scores; deciding on the share of cases passed absorbs part of that noise.

How to launch it. The intended operation is a test file with @pytest.mark.parametrize("c", CASES) on the function test_case and the command deepeval test run. In this session I did not run pytest: I called test_case in a loop, which exercises the same functions and the same metrics but not pytest's test collection. The code of deepeval test run that calls pytest and returns its exit code was read in the library's source, not observed running.

With a judge: the schema

When you move from metrics in code to model-based ones, little changes in form and much in risks. The listing is a schema: it defines a minimal judge by inheriting from DeepEvalBaseLLM, which requires four methods (load_model, generate, a_generate, get_model_name), and passes it to four standard metrics plus a GEval with a criterion in natural language. As for the table: ContextualPrecisionMetric and ContextualRecallMetric require the expected_output, therefore a golden dataset with references; faithfulness does not.

# b2.py - schema to adapt (not run): you need a judge model and its key or its endpoint
# Signatures checked with inspect.signature on deepeval 4.3.4; no model-based metric was computed.
from deepeval import assert_test
from deepeval.metrics import AnswerRelevancyMetric, ContextualPrecisionMetric, ContextualRecallMetric, FaithfulnessMetric, GEval
from deepeval.models import DeepEvalBaseLLM
from deepeval.test_case import LLMTestCase, SingleTurnParams

class MyJudge(DeepEvalBaseLLM):
    """Makes any model usable: these are the four abstract methods."""
    def load_model(self):
        return YOUR_MODEL_CLIENT
    def generate(self, prompt: str) -> str:
        return self.load_model().generate(prompt)             # replace with your client's call
    async def a_generate(self, prompt: str) -> str:
        return await self.load_model().generate_async(prompt)
    def get_model_name(self) -> str:
        return "name-of-your-judge"

judge = MyJudge()
caution = GEval(
    name="Cautious answer",
    criteria="The answer states its uncertainty when the context does not contain the requested information.",
    evaluation_params=[SingleTurnParams.INPUT, SingleTurnParams.ACTUAL_OUTPUT, SingleTurnParams.RETRIEVAL_CONTEXT],
    model=judge, threshold=0.7,
)
metrics = [
    FaithfulnessMetric(threshold=0.8, model=judge, include_reason=True),
    AnswerRelevancyMetric(threshold=0.7, model=judge),
    ContextualPrecisionMetric(threshold=0.7, model=judge),    # requires expected_output
    ContextualRecallMetric(threshold=0.7, model=judge),       # requires expected_output
    caution,
]

def test_rag(question, expected, rag):
    answer, contexts = rag(question)
    case = LLMTestCase(input=question, actual_output=answer, expected_output=expected, retrieval_context=contexts)
    assert_test(case, metrics)         # fails, with the reason for each metric, if one is below its threshold

Three choices to motivate. The thresholds (0.8 for faithfulness, 0.7 for the others) are placeholders: derive them from your first run on the golden dataset, not from a table of "recommended thresholds", because a value recommended in the abstract knows neither your judge nor your domain. The GEval criterion is written for a behaviour that the other metrics do not see, caution, and that is the reason it exists: the documentation describes it as a versatile metric to use alongside the RAG-specific ones. The last choice is the assert_test on the case: if a metric is below threshold, the assertion fails and the message reports the reason given by the judge, which is why include_reason is useful in CI: you read why a case failed without reopening the model.

In continuous integration

The command is one, and the workflow is little more than a frame. The schema below is for GitHub Actions; it was not run (no repository, no runner), and I only verified that the file is valid YAML. The versions of the actions are the latest releases read from the GitHub API on 2026-11-12: check yours.

# b3.sh - the command that a CI gate runs: it returns pytest's exit code
deepeval test run tests/test_rag_gate.py

# Schema to adapt (not run): a GitHub Actions workflow that runs the gate on every pull request
name: rag-quality-gate
on:
  pull_request:
    paths: ["rag/**", "tests/test_rag_gate.py", "golden/**"]
jobs:
  gate:
    runs-on: ubuntu-latest
    timeout-minutes: 20
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-python@v7
        with:
          python-version: "3.12"
      - run: pip install deepeval pytest
      - run: deepeval test run tests/test_rag_gate.py
        env:
          DEEPEVAL_TELEMETRY_OPT_OUT: "YES"

The paths filter is the choice with the most effect on cost: the gate runs only when the RAG code, the test or the golden dataset change, not on every edit of a documentation file. The timeout-minutes keeps a stuck judge from burning minutes forever. One problem remains that no file solves: every run with a model-based judge spends tokens and time, so keep two levels, a small deterministic subset on every change (the metrics in code from the first listing) and the full golden dataset with the model-based judge at a sparser cadence, for example before a release. The thresholds of the model-based gate must be decided with the statistics of episode 1: a margin narrower than the confidence interval of your sample produces random blocks.

To build the set of questions that the gate runs you need sampling and stratification criteria: you find them in the article on the golden dataset. For a compact version of a regression gate with an example corpus there is also RAG Evaluation Pipeline: golden dataset, which builds the same kind of mechanism from another angle.

What you can do now

You can choose between Ragas and DeepEval according to the point in the cycle where you need the number: a table of scores to explore, a test that fails to decide. You can write a custom metric with BaseMetric and use it in assert_test without calling any model, and you can turn the results of single cases into a decision on the set. You know that DeepEval's default judge is not neutral with respect to the data and that thresholds must be derived from your own baseline. The next step concerns the set of questions on which all this rests, in episode 4, and the retriever-only metrics to put in the gate, in episode 5.