01 - RAG Explained: Retrieval-Augmented Generation Fundamentals

RAG Explained: Retrieval-Augmented Generation Fundamentals

Large Language Models (LLMs) have transformed how we interact with information. GPT-4, Claude, Gemini, and Llama can write code, summarize documents, answer complex questions, and reason about abstract problems. Yet, they suffer from a fundamental flaw that severely limits their use in professional settings: hallucinations.

When an LLM does not know the answer, it does not say "I don't know." Instead, it generates plausible-sounding text that can be entirely fabricated, delivered with the same confidence as verified facts. In enterprise, legal, or medical contexts, this behavior is unacceptable. The most effective and mature solution to this problem is called Retrieval-Augmented Generation (RAG).

In this first article of the AI Engineering and Advanced RAG series, we will start from the ground up: what RAG is, how it works, why it solves the hallucination problem, and how to build a working RAG system from scratch. By the end, you will have a solid understanding of the entire architecture and be ready to dive deeper into each component in the following articles.

1. The Hallucination Problem in LLMs

To understand why RAG is necessary, we must first understand the problem it solves. Large Language Models are fundamentally probabilistic language models: given an input text, they predict the most likely sequence of tokens as a continuation. This architecture, built on the Transformer attention mechanism, produces remarkable results but has an intrinsic limitation: the model does not "know" anything in the human sense. It generates statistically plausible text based on patterns learned during training.

Hallucinations are not an occasional bug: they are a direct consequence of the architecture. When the model lacks sufficient information to answer, it does not return an error. Instead, it generates the most probable text continuation, which can be entirely fabricated. The model was trained to produce fluent and coherent text, not to be factually accurate.

Real-World Hallucination Examples

To grasp the severity of the problem, consider real-world scenarios where hallucinations have significant consequences:

• Legal domain: An LLM might cite non-existent court rulings or fabricate statute numbers with convincing formatting
• Medical domain: It could suggest incorrect drug dosages or document undocumented drug interactions
• Technical documentation: It might describe APIs with parameters that do not exist or features never implemented
• Customer support: It could invent company policies, return procedures, or warranties that do not exist

The data confirms the scope of the problem: hallucination rates vary significantly across domains, with specialized fields like medicine and law still exhibiting rates of 10-20% even with the most advanced models. RAG represents the most effective technique for mitigating this issue, with documented reductions of up to 71%.

2. What is RAG: Definition and Origins

Retrieval-Augmented Generation (RAG) is an architectural paradigm that combines an information retrieval system with a generative model (LLM) to produce answers grounded in real data. The concept was formalized in 2020 by Patrick Lewis and colleagues at Meta AI in the seminal paper "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks".

The core intuition is simple: instead of asking the model to "remember" all information from its training, we provide it with the relevant documents at generation time. This process is called grounding: anchoring the generation to concrete, verifiable sources.

TRADITIONAL LLM:
  User question ────────────────────> [LLM] ──> Answer
                                        |
                             Based ONLY on training data
                             Hallucination risk: HIGH

RAG (Retrieval-Augmented Generation):
  User question ──> [RETRIEVAL] ──> Relevant documents
                        |                    |
                        |                    v
                        └────────> [LLM + Context] ──> Answer with citations
                                        |
                             Based on REAL DATA retrieved
                             Hallucination risk: LOW
    

The concept of grounding is crucial: every claim in the generated answer can be traced back to a specific document in the knowledge base. This means:

• Answers are verifiable: you can check the source document
• Answers are updatable: updating the documents automatically updates the answers
• Answers are controllable: you decide which documents the system can consult
• Answers are traceable: you can include citations with source references

3. How RAG Works: The Complete Pipeline

A RAG system consists of two macro-phases: the indexing pipeline, which prepares documents once (or periodically), and the query pipeline, which handles user questions in real time.

3.1 Indexing Pipeline (Offline Phase)

The indexing phase transforms raw documents into a structure optimized for semantic search. It consists of four sequential steps:

[Source Documents]
    |  PDF, HTML, Markdown, CSV, databases, APIs, emails...
    v
[1. DOCUMENT LOADING]
    |  Raw text extraction from source formats
    |  Metadata preservation (author, date, title)
    v
[2. CHUNKING (Text Splitting)]
    |  Split into manageable fragments
    |  Strategies: fixed-size, semantic, recursive
    |  Overlap between chunks to preserve context
    v
[3. EMBEDDING]
    |  Transform each chunk into a numeric vector
    |  Models: OpenAI, Sentence Transformers, Cohere
    |  Typical dimensions: 384, 768, 1536, 3072
    v
[4. VECTOR STORE]
    |  Save vectors in a specialized database
    |  Indexing for fast search (HNSW, IVF)
    |  ChromaDB, Pinecone, Weaviate, Milvus, Qdrant
    v
[Knowledge Base Ready for Queries]
    

3.2 Query Pipeline (Online Phase)

When a user asks a question, the query pipeline kicks in to find the most relevant documents and generate a grounded answer:

[User Question]
    |  "How do I configure OAuth authentication in our app?"
    v
[1. QUERY EMBEDDING]
    |  The question is transformed into a vector
    |  Same model used during indexing!
    v
[2. SIMILARITY SEARCH]
    |  Find the most similar chunks in the vector store
    |  Metrics: cosine similarity, L2, dot product
    |  Returns the top-k results (typically k=3..10)
    v
[3. CONTEXT ASSEMBLY]
    |  Retrieved chunks are assembled into a context
    |  Ordered by relevance, deduplicated
    v
[4. PROMPT CONSTRUCTION]
    |  Build the prompt with context + question
    |  Template: "Based on the following documents, answer..."
    v
[5. LLM GENERATION]
    |  The model generates the answer based on context
    |  Can include citations to source documents
    v
[Grounded Answer + Citations]
    

4. Document Processing: Chunking and Preparation

Chunking is one of the most critical phases in the entire RAG pipeline. The quality of results depends largely on how documents are split. Chunks that are too large dilute the semantic signal and waste space in the LLM's context window. Chunks that are too small lose the context needed to be meaningful.

4.1 Fixed-Size Chunking

The simplest strategy: split text into blocks of fixed size (e.g., 500 tokens) with optional overlap between consecutive chunks.

Original document (1500 tokens):
"Lorem ipsum dolor sit amet... [1500 tokens of text]"

With chunk_size=500 and overlap=50:

Chunk 1: tokens 1-500     ────────────┐
Chunk 2: tokens 451-950   ──┐ overlap │
Chunk 3: tokens 901-1400  ──┘         │
Chunk 4: tokens 1351-1500             │
                           ────────────┘
The overlap ensures that context at chunk boundaries is not lost.
    

4.2 Recursive Character Splitting

A more sophisticated strategy that attempts to split while respecting document structure. It uses a hierarchy of separators: first tries paragraphs ("\n\n"), then lines ("\n"), then sentences (". "), then words (" "). This preserves semantic context better than fixed-size splitting.

4.3 Semantic Chunking

The most advanced strategy: uses embeddings themselves to determine where to split. It calculates similarity between consecutive sentences and creates a new chunk when similarity drops below a threshold, indicating a topic change. This produces variable-sized but semantically coherent chunks.

5. Embeddings: Transforming Text into Vectors

Embeddings are the mathematical heart of RAG. An embedding is a numeric representation (a vector of decimal numbers) that captures the semantic meaning of a text. Two sentences with similar meaning will have vectors that are "close" in multidimensional space, regardless of the words used.

INPUT: Text string
OUTPUT: Vector of decimal numbers (e.g., 1536 dimensions)

Example:
"The cat sleeps on the couch"   --> [0.23, -0.45, 0.67, 0.12, -0.89, ...]
"The feline rests on the sofa"  --> [0.22, -0.44, 0.68, 0.11, -0.88, ...]
                                      ^^ Very SIMILAR vectors (same meaning)

"The price of gold is rising"   --> [-0.56, 0.78, -0.12, 0.91, 0.34, ...]
                                      ^^ Very DIFFERENT vector (different meaning)
    

The embedding model is a neural network trained on massive amounts of text to learn semantic relationships between words, phrases, and concepts. It does not produce a merely syntactic representation (like bag-of-words or TF-IDF), but captures the deeper meaning of the text.

The choice of embedding model depends on the use case: larger models (3072 dimensions) offer a richer representation but cost more in terms of storage and computation. For many use cases, models with 768-1536 dimensions offer an excellent balance between quality and cost.

6. Vector Store: The Database for Embeddings

A vector store (or vector database) is a specialized database for storing, indexing, and searching high-dimensional vectors. Unlike traditional databases that look for exact matches (SQL WHERE), a vector store finds the vectors most similar to the query vector.

6.1 Similarity Metrics

Search in a vector store relies on distance/similarity metrics between vectors. The three most common metrics are:

6.2 Vector Database Overview

The vector database market has exploded with RAG adoption. Here is an overview of the main tools available:

To get started, ChromaDB is the simplest choice: it installs via pip, works in-memory or with disk persistence, and integrates natively with LangChain. For production, Pinecone (managed) and Qdrant (self-hosted) are among the most popular options.

7. Retrieval: Finding the Relevant Documents

The retrieval phase is where the user's question gets transformed into a vector and compared against all vectors in the store to find the most relevant chunks. This process happens in milliseconds even with millions of indexed documents, thanks to approximate nearest neighbor algorithms like HNSW (Hierarchical Navigable Small World).

Query: "How do I configure OAuth in our app?"
  |
  v
[1] Query Embedding ──> [0.34, -0.21, 0.56, ...]
  |
  v
[2] Similarity Search in Vector Store
  |   Compares query vector against all stored vectors
  |   Uses cosine similarity as metric
  |
  v
[3] Top-K Results (e.g., k=5)
  |
  |   Score: 0.92 - "OAuth 2.0 configuration for web apps..."
  |   Score: 0.87 - "Guide to OAuth authentication flows..."
  |   Score: 0.83 - "Setting up redirect URIs for OAuth..."
  |   Score: 0.76 - "Comparing OAuth and SAML for SSO..."
  |   Score: 0.71 - "API security with JWT tokens..."
  |
  v
[4] Filtering and Reranking (optional)
  |   Filter by metadata (date, category, source)
  |   Reranking with cross-encoder model
  |
  v
[Relevant Chunks Ready for the Prompt]
    

7.1 The Top-K Parameter

The top-k parameter determines how many chunks are retrieved. The choice is a trade-off:

• K too low (1-2): Risk of missing relevant information
• K too high (20+): Too much noise in context, wasted tokens, risk of confusing the model
• Optimal K (3-7): Good balance between coverage and precision

7.2 Relevance Scoring

Each result has a relevance score. Chunks with low scores should be filtered out because they add noise without value. A common threshold is 0.7 for cosine similarity: anything below gets discarded. In practice, the optimal threshold depends on the domain and should be calibrated experimentally.

8. Generation: Building the Answer with the LLM

The final phase of the RAG pipeline: retrieved chunks are assembled into a structured context and inserted into the LLM prompt alongside the user's question. The model generates the answer based exclusively (or primarily) on the provided context.

8.1 The Prompt Template

A good RAG prompt template must instruct the model to:

• Use only information from the provided context
• Acknowledge when it cannot find the answer in the documents
• Cite sources when possible
• Not fabricate information absent from the context

You are a technical assistant. Answer questions ONLY based on
the provided context. If the answer is not in the context,
reply "I could not find sufficient information in the available
documents."

CONTEXT:
---
{context}
---

QUESTION: {question}

INSTRUCTIONS:
1. Answer clearly and concisely
2. Cite the source document in square brackets [Source: doc_name]
3. If the context does not contain the answer, say so explicitly
4. Do not fabricate information not present in the context

ANSWER:

8.2 Citation Tracking

One of RAG's most valuable features is the ability to track citations. Each chunk in the context can be labeled with an identifier (e.g., [DOC-1], [DOC-2]) and the model is instructed to reference these identifiers in its answer. This allows users to verify every claim by consulting the original document.

9. Complete RAG Architecture: The Big Picture

Now that we have examined each component individually, let us assemble the complete end-to-end architecture of a RAG system:

                    OFFLINE PHASE (Indexing)
    ┌──────────────────────────────────────────────────────┐
    │                                                      │
    │  [Documents]  ──>  [Loader]  ──>  [Chunker]         │
    │   PDF, HTML        Text           Split into         │
    │   MD, CSV          extraction     fragments          │
    │                                       │              │
    │                                       v              │
    │                    [Embedding Model]                  │
    │                     Text ──> Vector                   │
    │                          │                           │
    │                          v                           │
    │                    [Vector Store]                     │
    │                     ChromaDB, Pinecone                │
    │                     Weaviate, Qdrant                  │
    │                                                      │
    └──────────────────────────────────────────────────────┘

                    ONLINE PHASE (Query)
    ┌──────────────────────────────────────────────────────┐
    │                                                      │
    │  [User Question]                                     │
    │       │                                              │
    │       v                                              │
    │  [Query Embedding] ──> [Similarity Search]           │
    │       Same model           Top-K chunks              │
    │       as indexing               │                    │
    │                                 v                    │
    │                    [Context Assembly]                 │
    │                     Chunks + Metadata                 │
    │                          │                           │
    │                          v                           │
    │  [Prompt Template + Context + Question]              │
    │                          │                           │
    │                          v                           │
    │                       [LLM]                          │
    │                    GPT-4, Claude                      │
    │                    Llama, Gemini                      │
    │                          │                           │
    │                          v                           │
    │              [Grounded Answer + Citations]            │
    │                                                      │
    └──────────────────────────────────────────────────────┘
    

10. Practical Example: Minimal RAG with Python

Let us build a working RAG system with the minimum code necessary. We will use LangChain as the orchestration framework, OpenAI for embeddings and generation, and ChromaDB as the local vector store.

10.1 Project Setup

# Create a virtual environment
python -m venv rag-env
source rag-env/bin/activate  # Linux/Mac

# Install dependencies
pip install langchain langchain-openai langchain-community
pip install chromadb
pip install pypdf  # For loading PDFs

10.2 Indexing Pipeline

import os
from langchain_community.document_loaders import (
    PyPDFLoader,
    TextLoader,
    DirectoryLoader
)
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import Chroma

# Configure API key
os.environ["OPENAI_API_KEY"] = "sk-..."

# 1. DOCUMENT LOADING
# Load all PDFs from a directory
loader = DirectoryLoader(
    "./documents/",
    glob="**/*.pdf",
    loader_cls=PyPDFLoader
)
documents = loader.load()
print(f"Loaded {len(documents)} documents")

# 2. CHUNKING
# Recursive splitting with overlap
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,       # 500 characters per chunk
    chunk_overlap=50,     # 50 characters overlap
    length_function=len,
    separators=["\n\n", "\n", ". ", " ", ""]
)
chunks = text_splitter.split_documents(documents)
print(f"Created {len(chunks)} chunks")

# 3. EMBEDDING + 4. VECTOR STORE
# Create embeddings and save to ChromaDB
embedding_model = OpenAIEmbeddings(
    model="text-embedding-3-small"
)

vectorstore = Chroma.from_documents(
    documents=chunks,
    embedding=embedding_model,
    persist_directory="./chroma_db",
    collection_name="company_documents"
)
print("Vector store created and persisted to disk")

10.3 Query Pipeline

import os
from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain_community.vectorstores import Chroma
from langchain.chains import RetrievalQA
from langchain.prompts import PromptTemplate

os.environ["OPENAI_API_KEY"] = "sk-..."

# Load the existing vector store
embedding_model = OpenAIEmbeddings(
    model="text-embedding-3-small"
)
vectorstore = Chroma(
    persist_directory="./chroma_db",
    embedding_function=embedding_model,
    collection_name="company_documents"
)

# Configure the retriever
retriever = vectorstore.as_retriever(
    search_type="similarity",
    search_kwargs={"k": 5}  # Retrieve top 5 similar chunks
)

# Define the prompt template
prompt_template = PromptTemplate(
    template="""You are an expert technical assistant.
Answer the question based ONLY on the provided context.
If you cannot find the answer in the context, say so explicitly.

CONTEXT:
{context}

QUESTION: {question}

ANSWER (with citations to source documents):""",
    input_variables=["context", "question"]
)

# Create the RAG chain
llm = ChatOpenAI(
    model="gpt-4o",
    temperature=0  # Deterministic for factual answers
)

rag_chain = RetrievalQA.from_chain_type(
    llm=llm,
    chain_type="stuff",  # Insert all chunks into the prompt
    retriever=retriever,
    return_source_documents=True,
    chain_type_kwargs={
        "prompt": prompt_template
    }
)

# Execute a query
result = rag_chain.invoke(
    {"query": "How do I configure OAuth authentication?"}
)

# Print the answer
print("ANSWER:")
print(result["result"])
print("\nSOURCES:")
for doc in result["source_documents"]:
    print(f"  - {doc.metadata.get('source', 'N/A')}"
          f" (page {doc.metadata.get('page', 'N/A')})")

ANSWER:
To configure OAuth authentication in our application,
follow these steps:
1. Register the app with the OAuth provider (Google, GitHub, etc.)
2. Configure redirect URIs in config.yaml [Source: oauth-guide.pdf]
3. Implement the Authorization Code flow [Source: auth-architecture.pdf]
...

SOURCES:
  - documents/oauth-guide.pdf (page 12)
  - documents/auth-architecture.pdf (page 5)
  - documents/security-faq.pdf (page 3)
      

10.4 Project Structure

rag-project/
  ├── documents/           # Your source documents
  │   ├── oauth-guide.pdf
  │   ├── architecture.pdf
  │   └── faq.pdf
  ├── chroma_db/           # Persistent vector store (generated)
  ├── indexing.py           # Indexing script
  ├── query.py              # Query script
  ├── requirements.txt      # Dependencies
  └── .env                  # API keys (never commit!)
    

11. Naive RAG vs Advanced RAG

The system we built so far is what the community calls Naive RAG: a linear, straightforward implementation that works surprisingly well for many use cases. However, it has limitations that Advanced RAG techniques aim to overcome.

11.1 Limitations of Naive RAG

• Query-Document Mismatch: The user's question might use different terms than the documents. "How do I reset my password?" vs document "Credential recovery procedure"
• Chunk Boundary Issues: Relevant information might be split across two consecutive chunks
• Lost in the Middle: LLMs tend to give more weight to chunks at the beginning and end of the context, neglecting those in the middle
• Single-Hop Limitation: Cannot answer questions requiring synthesis from multiple documents

11.2 Advanced RAG Techniques

NAIVE RAG:
  Query ──> Embedding ──> Search ──> Top-K ──> LLM ──> Answer

ADVANCED RAG:
  Query ──> [Query Analysis]
              │
              ├──> Query Rewriting ──> Embedding ──> Search ─┐
              ├──> HyDE Generation ──> Embedding ──> Search ─┤
              └──> Multi-Query ──────> Embedding ──> Search ─┘
                                                       │
                                        [Merge + Deduplicate]
                                               │
                                        [Reranker Model]
                                               │
                                        [Context Compression]
                                               │
                                            [LLM + Citations]
                                               │
                                        [Verified Answer]
    

We will dive deep into each of these techniques in the following articles of this series, particularly in the article on Hybrid Retrieval and Reranking.

12. When to Use RAG: Decision Framework

RAG is not the answer to every problem. There are three primary approaches to customizing LLM behavior, each with distinct advantages and limitations. The choice depends on your specific use case.

12.1 Practical Decision Tree

Need to customize an LLM?
│
├── Does data change frequently? (documents, policies, catalogs)
│   └── YES ──> RAG
│         (Data updates without re-training)
│
├── Need to cite sources?
│   └── YES ──> RAG
│         (Answer traceability)
│
├── Is the domain stable and well-defined?
│   ├── Do you have a large training dataset?
│   │   └── YES ──> Fine-Tuning
│   │         (Deep behavior customization)
│   └── NO ──> RAG or Prompt Engineering
│
├── Only need to change tone/format/style?
│   └── YES ──> Prompt Engineering
│         (No additional infrastructure)
│
└── Need complex reasoning on proprietary data?
    └── YES ──> RAG + Fine-Tuning (Hybrid Approach)
          (Best of both worlds)
    

12.2 Ideal Use Cases for RAG

• Enterprise chatbots: Answer questions based on internal documentation
• Semantic search: Find relevant documents by meaning, not just keywords
• Knowledge base Q&A: Dynamic FAQs that update with the documents
• Legal assistants: Answers grounded in regulations and case law
• Technical support: Resolutions based on past tickets and manuals
• Document analysis: Extract insights from reports, contracts, research papers

12.3 When NOT to Use RAG

• Creative tasks: Creative writing, brainstorming, idea generation (the model needs freedom)
• General conversation: Social chatbots where specific data is not needed
• Few-data tasks: If you only have a handful of documents, prompt engineering might suffice
• Ultra-low latency: If retrieval latency is unacceptable (under 50ms)

13. The RAG Market: Numbers and Trends

RAG is no longer an academic concept: it has become a production architecture adopted at scale. The market numbers confirm its strategic importance.

The sectors driving adoption are legal, medical, customer support, and financial services -- all domains where factual accuracy and source citability are non-negotiable requirements. The 2025-2026 trend is the shift from experimentation to large-scale production, with growing emphasis on compliance, monitoring, and data quality.

14. Conclusions and Next Steps

In this article we built a comprehensive understanding of RAG: from the problem it solves (LLM hallucinations) to the end-to-end architecture, passing through every component of the pipeline. We saw a working practical implementation and compared RAG with its alternatives (fine-tuning, prompt engineering).

Next Article: Embeddings and Semantic Search

In the next article of this series, we will dive deep into the most fascinating component of RAG: embeddings. We will explore how they work internally, how to choose the right model, how to evaluate their quality, and how to optimize semantic search. We will also cover evaluation metrics and benchmarking techniques to measure the performance of your retrieval system.