06 - Faithfulness metric: how to detect when LLM hallucinates given the context

Faithfulness: how far the answer stays inside the context

The retrieved context says: "The Base plan costs 100 euros a month". The model's answer says: "The Base plan costs 50 euros a month, with a 50% discount for the first year". Neither statement is supported by the document, yet the sentence is fluent and credible to anyone who doesn't have the document in front of them. The retriever did its job; it is the generator that rewrote the fact. This is hallucination with respect to the context, and faithfulness is the metric created to measure it without needing a reference answer.

Here you build the mechanism from scratch. First with a teaching verifier in Python that runs on your machine without any model, so you can see where errors come from and how to read them; then with the real tools (Ragas and an NLI classifier), shown as schemas to adapt because they need a model that is not run here. This is the sixth episode: for retrieval metrics see the episode on retrieval quality; for how these measures fit into a development cycle, see the episode on automation.

The definition, with an example you can redo by hand

The Ragas documentation defines it like this: an answer is faithful if all of its claims can be inferred from the retrieved context, and the score is the number of supported claims divided by the number of claims in the answer. Its example: asked where and when Einstein was born, the answer "in Germany on 20 March 1879" breaks down into two statements; the first is supported, the second is not (the context says 14 March), and the score is 1/2 = 0.5. Note that the wrong date costs half the score even though the other half is correct: the metric weighs claims, not severity.

The calculation always splits into the same four steps: break the answer into atomic claims, check each one against the context, divide the supported ones by the total, aggregate the scores over many answers. The first and second steps are the hard ones and are handed to a judge (an LLM or a classifier); the third and fourth are arithmetic.

• You don't need a golden dataset: question, retrieved context and generated answer are enough, so it can be computed on real traffic.
• It measures the answer-context relationship, not answer-reality: a true fact that is absent from the context lowers the score.
• It does not measure relevance: a faithful answer may not answer the question.

The second point is the most misunderstood. If the model adds an exact fact from its own training, faithfulness drops all the same, and that is intended: in a RAG system you care that every statement is anchored to a document you can show the user. For relevance you need other metrics, which in the series you see together in the episode on Ragas.

A teaching verifier: what it sees and what it cannot see

To understand where a metric goes wrong, it helps to build a transparent version of it. The listing below compares each claim with the sentences of the context: it picks the sentence that covers the most content words (five-letter stems) and numbers, then decides. If the claim's numbers are in the sentence and coverage exceeds 0.75, the claim is "supported". If the sentence covers the words but the numbers don't match, it is "contradicted". Otherwise it is "unverifiable". The lines that matter are the construction of content, which separates words from numbers, and the elif (n_c - n_f) branch, which tells a wrong number apart from missing information. It is not a language model: it understands neither synonyms nor negations.

import re

STOP = {"the", "a", "an", "of", "on", "in", "with", "and", "to", "is", "are", "it",
        "has", "have", "that", "include", "includes", "offer", "offers"}


def content(text):
    """Stems of the content words and set of numbers in a sentence."""
    words = [t for t in re.findall(r"[a-z0-9/]+", text.lower()) if t not in STOP]
    numbers = {t for t in words if any(c.isdigit() for c in t)}
    stems = {t[:5] for t in words if t not in numbers}
    return stems, numbers


def verify(claim, sentences, threshold=0.75):
    """Compare a claim with every context sentence and keep the closest one."""
    s_c, n_c = content(claim)
    best = None
    for sentence in sentences:
        s_f, n_f = content(sentence)
        coverage = len((s_c | n_c) & (s_f | n_f)) / len(s_c | n_c)
        if best is None or coverage > best[0]:
            best = (coverage, sentence, s_f, n_f)
    coverage, sentence, s_f, n_f = best
    word_cov = len(s_c & s_f) / len(s_c) if s_c else 1.0
    if n_c <= n_f and coverage >= threshold:
        label = "supported"
    elif (n_c - n_f) and word_cov >= threshold:
        label = "contradicted"
    else:
        label = "unverifiable"
    return label, round(coverage, 2), sentence


if __name__ == "__main__":
    CONTEXT = [
        "The Base plan costs 100 euros a month and includes 5 users.",
        "The Premium plan costs 200 euros a month, with unlimited users and 24/7 support.",
        "Phone support is not available on the Base plan.",
    ]
    ANSWER = ("The Premium plan costs 200 euros a month. It includes unlimited users. "
              "It offers 24/7 support. It has 100 GB of storage.")

    claims = [c.strip() for c in ANSWER.split(".") if c.strip()]
    supported = 0
    for claim in claims:
        label, coverage, sentence = verify(claim, CONTEXT)
        supported += label == "supported"
        print(f"{label:13s} cov={coverage:.2f}  {claim}")
        print(f"{'':13s} closest sentence: {sentence}")
    print(f"faithfulness = {supported}/{len(claims)} = {supported / len(claims):.2f}")

    wrong = "The Base plan costs 50 euros a month"
    print(wrong, "->", verify(wrong, CONTEXT)[:2])

# Output:
# supported     cov=1.00  The Premium plan costs 200 euros a month
#               closest sentence: The Premium plan costs 200 euros a month, with unlimited users and 24/7 support.
# supported     cov=1.00  It includes unlimited users
#               closest sentence: The Premium plan costs 200 euros a month, with unlimited users and 24/7 support.
# supported     cov=1.00  It offers 24/7 support
#               closest sentence: The Premium plan costs 200 euros a month, with unlimited users and 24/7 support.
# unverifiable  cov=0.33  It has 100 GB of storage
#               closest sentence: The Base plan costs 100 euros a month and includes 5 users.
# faithfulness = 3/4 = 0.75
# The Base plan costs 50 euros a month -> ('contradicted', 0.83)

The result is the 3/4 of the formula. The first three claims are anchored to the Premium plan sentence; the fourth, "It has 100 GB of storage", lands on the Base plan sentence only because it shares the number 100, with coverage 0.33: the verifier declares it unverifiable, and that is correct. This is also a warning: the "100" attached itself to the wrong sentence, and a slightly less cautious verifier would have confused the price with the disk space. The last line shows the other outcome: "The Base plan costs 50 euros a month" shares almost everything with the Base sentence and only the number changes, so it is marked contradicted, not simply unverifiable.

The distinction between the two outcomes matters in production. An unverifiable claim may be a tolerable omission or useful information that is not in the documents; a contradicted claim is an active hallucination, the kind that leads a user to decide on a wrong price. If you record both counters, you can put the alarm on the second and keep the first as an indicator of corpus coverage.

From claim to document: attribution and degrees of contradiction

Knowing that a claim is contradicted doesn't tell you where to look. For debugging you need two more pieces of information: which chunk the model should have used, and whether the context itself is inconsistent. The second listing computes the first with cosine similarity on TF-IDF (scikit-learn) between the claim and each chunk, and the second by counting in how many chunks the verifier says "contradicted": none is NONE, all is ALL, some is SOMETIMES. The context is built on purpose with two price lists from different years.

from a_verifier import verify
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity

CHUNKS = [
    "2025 price list: the Base plan costs 100 euros a month.",
    "2026 price list: the Base plan costs 120 euros a month.",
    "Phone support is not available on the Base plan.",
]


def level(claim, chunks):
    """NONE / SOMETIMES / ALL: how many chunks contradict the claim."""
    labels = [verify(claim, [c])[0] for c in chunks]
    n = labels.count("contradicted")
    return ("NONE" if n == 0 else "ALL" if n == len(chunks) else "SOMETIMES"), labels


claim = "The Base plan costs 120 euros a month"
lvl, labels = level(claim, CHUNKS)
print(lvl, labels)

# Attribution: which chunk is most similar to the claim (cosine similarity on TF-IDF)
vec = TfidfVectorizer().fit(CHUNKS + [claim])
sim = cosine_similarity(vec.transform([claim]), vec.transform(CHUNKS))[0]
for s, c in sorted(zip(sim, CHUNKS), reverse=True):
    print(f"{s:.2f}  {c}")

# Output:
# SOMETIMES ['contradicted', 'supported', 'unverifiable']
# 0.74  2026 price list: the Base plan costs 120 euros a month.
# 0.54  2025 price list: the Base plan costs 100 euros a month.
# 0.19  Phone support is not available on the Base plan.

The claim "costs 120 euros" is backed by the 2026 price list and contradicted by the 2025 one: level SOMETIMES. The fault is not the generator's, which picked the right line, but the corpus's, which keeps two versions of the same fact. A judge that looked at a single chunk would give 1.0 or 0.0 depending on which chunk it received. Attribution confirms the reading: the chunk most similar to the claim, at 0.74, is exactly the 2026 price list. The remedy lies upstream, in document version management, not in the prompt.

Aggregating and alarming: the mean hides the tail

One score per answer is not enough; you need an overall view and a rule that says when to worry. The third listing shows two things. In the first part, eight answers with a mean of 0.781 nevertheless include one answer at 0.0 and one at 0.5: the standard deviation 0.364 and the share below 0.5 (12.5%) tell you what the mean hides. In the second, a comparison between the latest evaluation and the mean of the previous four triggers a warning when the drop exceeds 5% and a critical alarm beyond 10%.

import numpy as np


def summary(scores):
    p = np.array(scores)
    return {"mean": p.mean(), "std_dev": p.std(ddof=1),
            "p10": np.percentile(p, 10), "below_0.5": float((p < 0.5).mean())}


def alarm(history, window=4, warning=0.05, critical=0.10):
    """Compare the latest score with the mean of the `window` previous evaluations."""
    if len(history) < window + 1:
        return None
    base = float(np.mean(history[-(window + 1):-1]))
    drop = (base - history[-1]) / base
    if drop >= critical:
        return f"CRITICAL drop {drop:.1%} (base {base:.3f}, now {history[-1]:.2f})"
    if drop >= warning:
        return f"WARNING drop {drop:.1%} (base {base:.3f}, now {history[-1]:.2f})"
    return None


# Eight answers evaluated in one evening: the mean hides the answer scoring 0.0
per_answer = [1.0, 1.0, 0.75, 1.0, 0.0, 0.5, 1.0, 1.0]
for k, v in summary(per_answer).items():
    print(f"{k:10s} {v:.3f}")

# Weekly history invented for teaching purposes
history = [0.88, 0.86, 0.89, 0.87, 0.85, 0.84, 0.82, 0.76]
for week in range(5, len(history) + 1):
    print(f"week {week}:", alarm(history[:week]) or "ok")

# Output:
# mean       0.781
# std_dev    0.364
# p10        0.350
# below_0.5  0.125
# week 5: ok
# week 6: ok
# week 7: ok
# week 8: CRITICAL drop 10.1% (base 0.845, now 0.76)

The weekly history is invented for teaching purposes, and the 5% and 10% thresholds are values to calibrate, not standards: measure the natural variability of your metric over repetitions of the same set and set the alarm beyond that fluctuation. In the example, week 8 falls by 10.1% from the 0.845 base, and the critical alarm fires with a minimal margin: a threshold at 0.11 would have stayed silent. For the full monitoring cycle in operation, from sampling to the trigger, I refer you to episode 9 and, for the general picture of RAG in production, to this deep dive from the AI Engineering series.

With the real tools: Ragas and an NLI classifier

The teaching verifier is lexical; the real tools are semantic. Ragas hands both the breakdown into claims and the verification to a judge LLM. In the current documentation the metric is imported from ragas.metrics.collections and called with ascore (or score synchronously); for the earlier interface based on evaluate, the documentation announces deprecation from version 0.4 and removal from 1.0. The schema is not executed here: it needs a client for a judge model, and the score depends on the judge you choose.

# Schema to adapt, not executed: it needs a client for a judge model.
# "collections" API from the Ragas documentation (for the old evaluate(...) the
# documentation announces deprecation from 0.4 and removal from 1.0).
from ragas.llms import llm_factory
from ragas.metrics.collections import Faithfulness

llm = llm_factory("<judge-model-name>", client=client)  # compatible async client
scorer = Faithfulness(llm=llm)

result = await scorer.ascore(
    user_input="How much does the Premium plan cost?",
    response="The Premium plan costs 200 euros a month. It has 100 GB of storage.",
    retrieved_contexts=["The Premium plan costs 200 euros a month, with unlimited users."],
)
print(result.value)   # expected: 0.5 (one claim out of two supported), but it depends on the judge

The cheaper alternative is a textual inference (NLI) classifier. The model cross-encoder/nli-deberta-v3-base receives the (premise, hypothesis) pair and returns three scores in the order contradiction, entailment, neutral; its model card declares training on SNLI and MultiNLI and an accuracy of 92.38 on SNLI-test and 90.04 on MNLI mismatched. These are English data: for Italian texts the result is not guaranteed and must be validated on your own examples. The apply_softmax parameter of predict defaults to False, so without enabling it you get logits, not probabilities, and a threshold such as 0.5 loses meaning.

# Schema to adapt, not executed: it downloads a model from Hugging Face.
from sentence_transformers import CrossEncoder

model = CrossEncoder("cross-encoder/nli-deberta-v3-base")
pairs = [  # (premise = context, hypothesis = claim); the model is trained on English
    ("The Base plan costs 100 euros a month.", "The Base plan costs 50 euros a month."),
    ("The Base plan costs 100 euros a month.", "The Base plan costs 100 euros a month."),
]
scores = model.predict(pairs, apply_softmax=True)   # one row per pair, three columns
order = ["contradiction", "entailment", "neutral"]  # order declared in the model card
labels = [order[row.argmax()] for row in scores]

The trade-off between the two is this: the generative judge explains its verdict but is slow and non-deterministic; the classifier is deterministic and suited to mass checking, but produces no explanations and has a specific training domain. In practice you use them in cascade, the classifier as a filter and the judge on the doubtful cases, and I cannot give you a time per claim here: it depends on the machine and has to be measured.

Where the metric goes wrong

Four blind spots explain almost every case in which faithfulness flags a problem that isn't there, or doesn't see one that is.

• Paraphrase: "thousands of users" against "10,000 concurrent users" confuses a lexical verifier and sometimes an NLI one too.
• Implicit information: "studied computer science" is implied by "graduate in computer science" but not identical.
• Negations: "doesn't use Docker" and "Docker is not used" are equivalent, but models treat them unevenly.
• Claims that are too short: under five words they tend to come out neutral for lack of context.

The common remedy is not to trust a single signal: widen the context to adjacent sentences, compare several chunks and check a sample of borderline cases by hand. Before putting a judge into production, build fifty or sixty examples of your own with paraphrases and negations and measure how often it gets them wrong; it is a check that costs an afternoon and tells you whether the metric is usable in your domain.

Frequently asked questions

What exactly does faithfulness measure?

The share of the answer's claims supported by the retrieved context, that is, supported claims divided by total claims. It measures neither absolute quality nor relevance to the question, only fidelity to the context provided. A value of 0.9 means that one claim in ten cannot be verified from the context.

Can I compute it without a golden dataset?

Yes, and the Ragas paper places it among the metrics that need no human reference annotations. The triple question, context, answer is enough, so it can be computed on real traffic, provided you save the context actually passed to the model.

Which threshold should I use?

No threshold holds in general. Measure the metric's variability on your evaluation set by repeating the measurement several times, then set the alarm beyond that natural fluctuation. For high-risk domains (medical, legal) the threshold should be raised, but the value is decided by the cost of an error in your case.

When should I use an LLM judge and when an NLI classifier?

The classifier for screening many samples, the judge for doubtful cases or when you need an explanation. Both must be validated on your language and your domain before you trust the scores.

Where does evaluation belong in the flow?

Outside the request path: sample a share of the interactions, evaluate them in batch and write the score to a history. A judge in the critical path would add unpredictable latency to every answer.

What you can do now

You can compute faithfulness by hand, tell an unverifiable claim from a contradicted one, trace from the claim back to the chunk that should have supported it, recognize an inconsistent context and set an alarm that doesn't rely on the mean alone. You also know which parts of the real system require a model and which precautions to take before trusting a classifier trained on another language. In the next episode, the automation of evaluation, you see how to run all this in batch without waiting hours.