RAG Evaluation Pipeline: golden dataset Italian

1. Problem: how to measure RAG quality?

Prompt: "Explain Italian administrative law". Correct vs incorrect response is not binary. Quality dimensions:

• Faithfulness: Is response coherent with retrieved documents? If retrieval = [doc1, doc2], response must NOT contain information outside [doc1, doc2] (hallucination check).
• Relevance: Are retrieved documents pertinent to query? E.g. "How to open company?" retrieves "VAT Tax 2026" → relevance low.
• Completeness: Does response cover all query aspects? E.g. "What are rights and duties?" must include both, not just rights.
• Accuracy: Are factual information correct? Requires fact-checking vs trusted sources (e.g. official Tax Authority website).

2. RAGAS framework: core metrics

RAGAS (Retrieval-Augmented Generation Assessment) is 2023 framework from AIPlus, became de-facto standard 2024-2026.

4 primary metrics:

• Faithfulness [0, 1]: "Is response supported by retrieval?" LLM evaluates how many claims in response appear in docs. Formula: claims_supported / claims_total. Threshold ≥0.85 for production.
• Answer Relevance [0, 1]: "Does response answer the query?" LLM extracts implicit query from response, calculates cosine-similarity vs original query. Threshold ≥0.75.
• Context Precision [0, 1]: "Among top-10 retrieved docs, how many are relevant?" Ranking metric. Threshold ≥0.70.
• Context Recall [0, 1]: "Were all necessary documents retrieved?" Requires pre-annotated reference documents. Threshold ≥0.80.

3. Golden dataset Italian: 100 q/a pair template

Creating golden dataset (reference of truth) is foundational. Template for Italian dataset:

• Minimum 500 examples (sweet spot for evaluation)
• Domain balance: 30% civil law, 30% tax, 20% privacy, 20% other
• Manual validation: 10% samples review for quality
• No PII (real names, tax IDs) for GDPR compliance
• Source traced (for compliance audit)

4. Continuous evaluation loop

Production RAG requires weekly evaluation against golden dataset:

• Run RAGAS metrics on random 100 production queries
• Compare vs baseline golden metrics
• Alert if deterioration >5% on any metric
• Investigate root cause (retrieval degradation? model update? embedding drift?)
• Document findings in compliance log

Conclusion

RAGAS framework + Italian golden dataset is foundation for production RAG 2026. Automated continuous evaluation prevents silent failure. Cost: €0 if LLM on-prem, €0.05-0.10/evaluation if OpenAI.