What is Retrieval-Augmented Generation and why use it?
Retrieval-Augmented Generation (RAG) is a pattern where you fetch relevant documents from an external knowledge store at query time and pass them to a language model as context, so it answers from that evidence instead of only its training data. You use it when answers must reflect current, private, or domain-specific information, when you need citations, or when retraining the model is too slow or expensive. RAG also reduces hallucination because the model is told to ground its answer in the supplied passages.
When should I use RAG instead of fine-tuning?
Use RAG when the knowledge changes often, must be cited, or is private, because you can update an index in seconds without touching the model. Use fine-tuning when you need to change the model's style, format, or task behaviour rather than its facts, since fine-tuning teaches how to respond, not what is currently true. The two are complementary: a fine-tuned model can still retrieve facts via RAG, and many production systems do both.
How big should my chunks be?
There is no universal size, but a common starting point is roughly 200 to 500 tokens (about 300 to 600 characters per passage in many corpora) with a small overlap so context is not cut mid-thought. Larger chunks preserve more context but dilute relevance and waste the context window; smaller chunks are precise but can lose the surrounding meaning. The right size depends on your documents and model, so treat it as a parameter to tune against your eval set rather than a fixed rule.
Why combine keyword search with vector search?
Vector (semantic) search finds passages by meaning even when wording differs, but it can blur exact identifiers, codes, and rare terms; lexical search like BM25 nails those exact matches but misses paraphrases. Combining them as hybrid search and fusing the results (commonly with Reciprocal Rank Fusion) usually beats either method alone because each covers the other's blind spot. This is one of the highest-leverage, lowest-risk improvements you can make to a retriever.
What is re-ranking and do I need it?
Re-ranking is a second pass that re-scores your initial candidate set with a more accurate but slower model, typically a cross-encoder that reads the query and each passage together, then keeps only the best few for the generator. You need it when first-stage retrieval returns roughly relevant but poorly ordered results, which is common at larger top-k. It noticeably improves precision at the top of the list, at the cost of extra latency, so retrieve a wider net (for example top-50) and re-rank down to top-5.
How do I stop my RAG system from hallucinating?
Hallucination in RAG usually comes from weak retrieval or a prompt that lets the model fall back on memory, so attack both. Improve retrieval (hybrid search, re-ranking, better chunking) so the right evidence is actually present, and use a grounding prompt that instructs the model to answer only from the supplied context and to say "the context does not cover this" when it cannot. Then measure faithfulness on an evaluation set so regressions are caught before release rather than by users.
How do I evaluate a RAG pipeline?
Build a golden set of questions, each with a reference answer and the contexts that support it, then score your pipeline on retrieval and generation metrics separately. Retrieval is measured with context recall and context precision (did we fetch the right passages?), and generation with faithfulness and answer relevancy (did the model answer correctly from them?). Frameworks like RAGAS, promptfoo, and DeepEval automate this, and you should wire it into CI so a quality floor blocks bad merges.
Why does my RAG system retrieve irrelevant chunks?
The usual causes are poor chunking (passages that mix several topics or are cut mid-idea), a query embedding that does not match how the documents are phrased, or missing metadata filters that let unrelated content in. Diagnose by inspecting the actual retrieved chunks for failing questions rather than only reading the final answer. Fixes include structure-aware chunking, query expansion or HyDE for short queries, adding hybrid search, and filtering by source, language, or date.
How many chunks should I send to the model (what top-k)?
Top-k trades recall against noise and cost: too few and you miss supporting evidence, too many and you bury the answer in clutter while spending tokens. A common pattern is to retrieve a wide candidate set, re-rank, and then pass only the top 3 to 8 passages to the generator. Tune the final k on your eval set, watching faithfulness and answer relevancy rather than guessing.
What is prompt injection and how do I defend against it in RAG?
Prompt injection is when malicious instructions hidden inside retrieved documents or user input trick the model into ignoring its system prompt or taking unintended actions, and it is especially dangerous in RAG because untrusted text enters the prompt as context. Defend by treating retrieved content as data rather than commands, sanitising and clearly delimiting context, limiting the model's tools and privileges, and validating outputs before acting on them. Never let retrieved text alone authorise a sensitive action.
Which vector database should I choose?
Choose based on scale, your existing stack, and operational appetite rather than benchmarks alone. If you already run PostgreSQL, pgvector keeps everything in one database and is excellent up to millions of vectors; Qdrant, Weaviate, and Milvus are purpose-built for very large or high-throughput workloads; Redis with RediSearch suits low-latency, cache-adjacent use; managed services like Pinecone remove ops at the cost of lock-in. Most teams should start with the simplest option that fits and migrate only when a real limit appears.
How do I keep my RAG index fresh as documents change?
Make ingestion incremental and idempotent so re-running it only updates what changed, keyed by a stable document id and a content hash or version. Detect changes at the source (webhooks, change feeds, or scheduled diffs), re-chunk and re-embed only affected documents, and delete chunks for removed content. Pair this with cache invalidation tied to the same keys so stale embeddings and retrieval results do not linger.
Can RAG answer multi-step or multi-document questions?
Single-shot RAG, which retrieves once and answers, often fails on questions that need bridging facts spread across documents. For those, use multi-hop or agentic RAG, where the model retrieves, reads, reformulates, and retrieves again in a bounded loop, or decompose the question into sub-questions and merge their evidence. GraphRAG helps when the question needs corpus-wide structure or relationships rather than isolated passages. The trade-off is added latency and complexity, so reserve it for queries that genuinely need it.
What metrics matter for a production RAG service?
Track quality and operations together. For quality, monitor faithfulness, answer relevancy, and retrieval recall/precision on a rolling eval and on sampled live traffic; for operations, watch p95 latency, cost per query, retrieval and generation error rates, and the rate of "I don't know" responses. A rising "no answer" rate or falling recall often signals stale or insufficient content before users complain.
How do I reduce the cost and latency of a RAG system?
Cache aggressively: reuse embeddings for unchanged text, cache retrieval results for repeated queries, and use prompt/KV caching to reuse a stable grounding prefix across calls. Retrieve a smaller, re-ranked top-k so you send fewer tokens to the generator, choose an embedding model sized to your accuracy needs, and pick an ANN index tuned for your latency target. Measure cost and p95 latency per query so each optimisation is verified rather than assumed.
Do I always need a vector database for RAG?
No. For small or static corpora, an in-memory index or even brute-force cosine over a few thousand vectors is simple and fast enough, and a relational database with full-text search can serve lexical-only RAG. You need a dedicated vector store once the corpus grows to where approximate nearest-neighbour search and metadata filtering at scale matter for latency. Start with the simplest thing that meets your recall and latency targets and add infrastructure only when a real limit forces it.