Ground every answer and allow an explicit "I don't know"
A RAG system's value is that answers are traceable to sources, so the prompt must instruct the model to answer only from the supplied context and to state plainly when the context is insufficient. Without an escape hatch, the model fills gaps from memory and hallucinates confidently. Do: include "If the context does not contain the answer, say so" in the system prompt. Don't: leave the model free to "be helpful" by guessing when retrieval returns nothing relevant.
Chunk on structure, not arbitrary character counts
Splitting purely by a fixed character window cuts sentences and tables mid-idea and produces passages that mix unrelated topics, both of which wreck retrieval. Prefer structure-aware chunking on headings, sections, or sentence boundaries, with a small overlap to preserve continuity. Do: split a Markdown doc on its headings, then sub-split long sections. Don't: blindly slice every document into 1000-character blocks regardless of where the boundaries fall.
Use hybrid retrieval by default
Vector search captures meaning but blurs exact identifiers, codes, and rare terms; lexical BM25 nails those but misses paraphrases. Running both and fusing the results (Reciprocal Rank Fusion is a robust default) reliably beats either alone for little extra effort. Do: combine BM25 and vector hits with RRF. Don't: ship vector-only retrieval and then wonder why exact part numbers or error codes are never found.
Retrieve wide, then re-rank narrow
First-stage retrieval is fast but imprecise, so casting a wide net and re-ranking with a cross-encoder gives far better top results than trusting raw vector order. Pass only the re-ranked best few to the generator to keep the context tight. Do: retrieve top-50, re-rank, send top-5. Don't: send the raw top-20 vector hits straight to the model and pay for tokens on mostly-irrelevant passages.
Evaluate retrieval and generation separately
Bundling everything into one "answer looks good" judgement hides where failures come from. Measure retrieval with context recall and precision and generation with faithfulness and answer relevancy, so you know whether to fix the retriever or the prompt. Do: maintain a golden eval set and score the two stages independently. Don't: tune prompts to mask a retriever that never fetched the right passage.
Gate releases on an eval set in CI
Manual spot-checking does not catch regressions, and RAG quality drifts as data, models, and prompts change. A golden eval set scored automatically with a faithfulness floor turns quality into a merge gate. Do: fail CI when mean faithfulness drops below your calibrated threshold. Don't: edit the golden answers to make a regressed system pass, which defeats the entire purpose of the set.
Always filter by tenant, access level, and freshness
Retrieval that ignores who is asking can leak one customer's data into another's answer or surface outdated or unauthorised content. Enforce mandatory metadata filters (tenant, ACL, language, date) on every query, applied alongside the similarity search. Do: make the tenant filter non-optional in the retriever interface. Don't: rely on a "we'll add the filter in the application layer later" promise that one code path inevitably forgets.
Make ingestion incremental and idempotent
Full re-indexing on every change is slow, expensive, and a source of duplicates and drift. Key chunks by a stable document id plus a content hash so re-running ingestion updates only what changed and deletes chunks for removed content. Do: skip unchanged documents by comparing content hashes and upsert by external id. Don't: append on every run and accumulate stale, duplicated chunks that pollute retrieval.
Treat retrieved content as untrusted data, never as instructions
Documents in your corpus can contain text that tries to override the model's instructions (prompt injection), and in RAG that text lands directly in the prompt. Clearly delimit context, instruct the model to ignore instructions found inside it, limit the tools and privileges the model can use, and validate outputs before acting. Do: wrap context in explicit markers and keep sensitive actions behind separate authorisation. Don't: let a retrieved document's "ignore previous instructions" line change system behaviour or trigger a real action.
Add chunk-level context before embedding
An isolated chunk often loses the document context that makes it findable ("the rate" means nothing without "the 2026 import tariff rate"). Prepending a short chunk-specific summary of its source before embedding (contextual retrieval) measurably improves recall. Do: generate a one-line context header per chunk at ingestion time. Don't: embed bare fragments stripped of every clue about which document and section they came from.
Keep generation deterministic and cite sources
Grounded answers should be reproducible and verifiable, so run the generator at a low temperature and require it to attribute claims to the passages it used. This makes outputs stable across runs and lets users check the evidence. Do: set temperature near 0 and render citations to each source URL. Don't: use a high temperature for factual answers and ship un-cited claims users cannot verify.
Cache deliberately with explicit invalidation
Embeddings, retrieval results, and generated answers are all cacheable and can cut cost and latency dramatically, but a cache without invalidation serves stale answers after the source changes. Tie each cache's lifetime to its source's freshness, keyed so updates evict the right entries. Do: invalidate embedding and retrieval caches by the same document id/hash used in ingestion. Don't: cache final answers indefinitely and keep serving last quarter's policy after it was revised.
Inspect the actual retrieved chunks when debugging
When an answer is wrong, the root cause is far more often bad retrieval than bad generation, but you cannot see that by reading the answer alone. Log and review the exact chunks retrieved for failing questions to tell a recall failure from a generation failure. Do: capture query, retrieved chunk ids, and scores for every request you debug. Don't: keep rewriting the prompt to fix answers when the right passage was never retrieved in the first place.