What is BM25 and how it works

BM25 (Best Matching 25) is the most widely used ranking algorithm in modern information retrieval. It's a probabilistic formula developed in the 1990s that calculates the relevance of a document to a query based on term frequency.

The algorithm considers three main factors:

• Term frequency (TF): how many times the search term appears in the document
• Inverse document frequency (IDF): how many documents contain the term (rare terms are more relevant)
• Document length: BM25 normalizes results to prevent long documents from always getting higher scores

The basic formula is:

score(D, Q) = Σ IDF(qi) * (f(qi, D) * (k1 + 1)) / (f(qi, D) + k1 * (1 - b + b * |D| / avgdl))

Where k1 and b are tuning parameters (typically k1=2.0, b=0.75).

BM25 is extremely effective for queries with specific terms and well-defined keywords. If you search for "Canon DSLR camera", BM25 will find documents containing exactly these terms. It's fast, interpretable, and requires no training.

Dense embeddings and semantic search

Dense embeddings are numerical representations of text that capture semantic meaning. An embedding model converts a sentence (or word, or document) into a vector of numbers in a high-dimensional vector space.

Unlike BM25 which works with discrete tokens, embeddings capture conceptual relationships. For example:

• "automobile" and "car" will have very similar embeddings (low cosine distance)
• "cat" will have an embedding closer to "animal" than to "refrigerator"
• "Rome" and "Capital" will have a reduced cosine distance in semantic space

Search with embeddings works like this:

1. Convert the query into an embedding using the same model
2. Calculate similarity (usually cosine) between the query embedding and indexed document embeddings
3. Return documents with the highest similarity

Common embedding models include Sentence Transformers (all-MiniLM-L6-v2, all-mpnet-base-v2), OpenSearch Neural Search, and large language model embeddings. In the context of on-premises systems like Ollama, you can use models like nomic-embed-text or sentence-transformers.

Why combine BM25 and dense embeddings

Neither approach is perfect in isolation. BM25 excels at finding exact matches, but fails when language differs. Dense embeddings understand meaning, but can be imprecise with specific queries and require more computational resources.

Hybrid search leverages the strengths of both:

• BM25 for exact matches and specific terms (usernames, product codes, numbers)
• Dense embeddings for understanding meaning and finding semantically similar documents
• Weighted combination to get the best of both worlds

A study across various retrieval workloads demonstrated that hybrid search on average outperforms both pure BM25 and pure embeddings by approximately 15-30% in terms of NDCG@10 (Normalized Discounted Cumulative Gain).

Combination strategies

There are several techniques for merging BM25 and dense embedding results:

1. Simple ranking fusion

Run both searches (BM25 and embedding) and combine results with a weighted average. For each document, calculate a final score:

final_score = w1 * normalized_bm25_score + w2 * normalized_embedding_score

where w1 and w2 are weights (typically 0.5 + 0.5, but you can tune them for your use case).

2. Reciprocal Rank Fusion (RRF)

If BM25 returns the document at rank 1 and embeddings return the same document at rank 3, the RRF score will be:

RRF_score = 1/60 + 1/63 ≈ 0.0317

RRF is robust and doesn't require explicit normalization of heterogeneous scores. It's particularly useful when the two systems have very different score scales.

3. Two-stage re-ranking

Use BM25 as a fast first filter (retrieve top 100 documents), then re-rank with embeddings. This reduces embedding computation.

Implementation with RediSearch and Ollama

If you use Redis as a vector store (RediSearch 2.4+), hybrid search is native:

import redis
import numpy as np
from sentence_transformers import SentenceTransformer

# Connect to Redis
r = redis.Redis(host='localhost', port=6379, decode_responses=True)

# On-premise embedding model (e.g., Ollama with nomic-embed-text)
model = SentenceTransformer('model-name')

# Hybrid RediSearch index
schema = [
    {"name": "title", "type": "TEXT"},
    {"name": "content", "type": "TEXT"},
    {"name": "embedding", "type": "VECTOR", "algorithm": "HNSW", "dim": 768, "distance_metric": "cosine"}
]

# Hybrid query
query_text = "best configuration for RAG"
query_embedding = model.encode(query_text)

# BM25 part
bm25_results = r.execute_command(
    'FT.SEARCH', 'idx:hybrid',
    f'(@title|@content):("{query_text}")',
    'RETURN', '2', 'title', 'content',
    'LIMIT', '0', '10'
)

# Vector search part
vector_results = r.execute_command(
    'FT.SEARCH', 'idx:hybrid',
    f'*=>[KNN 10 @embedding $vec]',
    'PARAMS', '2', 'vec', query_embedding.tobytes(),
    'RETURN', '2', 'title', 'content'
)

# Fusion: combine results
combined = merge_results(bm25_results, vector_results, weights=(0.4, 0.6))

The key is using the `*=>[KNN ...]` syntax in RediSearch for vector search combined with FT.SEARCH for BM25.

Trade-offs and practical considerations

Latency

Hybrid search is slower than BM25 alone (BM25 ≈ 1-5ms, embeddings ≈ 10-50ms, hybrid ≈ 15-60ms). If latency is critical (e.g., real-time search), consider:

• Caching embeddings of frequent queries
• Re-ranking: use BM25 to filter quickly, then embeddings only on top N results
• Parallelizing BM25 and vector search requests

Index size

Embeddings require significant storage. A 768-dim model with 1 million documents requires approximately 3GB of RAM in Redis (768 float32 * 1M * 4 bytes). If storage is a constraint, consider smaller embeddings (384-dim) or quantized compression.

Freshness

BM25 is instantaneous after indexing. Embeddings have lag if batch indexing is asynchronous. For real-time updates, index both BM25 and embeddings in a transaction.

Tuning weights

Weights w1 and w2 depend on your use case. Use a validation set and evaluate NDCG, MRR (Mean Reciprocal Rank) or other ranking metrics. Typically:

• Dataset with very specific queries (codes, numbers): w1=0.7, w2=0.3
• Conversational dataset: w1=0.3, w2=0.7
• Balanced: w1=0.5, w2=0.5

Real-world use cases

RAG (Retrieval-Augmented Generation)

Hybrid search is now standard in RAG systems. For a natural language query, embeddings capture intent, while BM25 finds documents with specific terms (entity names, technical data). The generative model receives the most relevant context.

E-commerce

Users search products with vague queries ("comfortable blue jacket") and exact ones (SKU, brand). BM25 handles SKU and brand, embeddings understand "blue" vs "light blue". Result: fewer product-not-found errors.

Internal knowledge base

FAQ, technical documentation, procedures. BM25 finds the issue number or exact title, embeddings find semantically similar content even if written differently.

Question Answering

A Q&A system combining BM25 and embeddings reduces "no matching answer" and improves ranking accuracy of candidates.

Tools and libraries

• RediSearch 2.4+: Native support for hybrid search with `FT.SEARCH` + vector search
• Elasticsearch 8.x: `dense_vector` plugin for embeddings + native BM25, fusion with `rank fusion`
• Milvus 2.x: Vector database with HNSW, BM25 integration via sparse + dense vectors
• Weaviate: Support for native hybrid BM25 and vector search
• Qdrant: Modern vector database with hybrid search via fusion strategy
• Sentence Transformers: Python library for generating embeddings (Apache 2.0)
• PyMilvus, Redis-py, Elasticsearch-py: Python clients for indices

Evaluation metrics

To evaluate hybrid search quality, use:

• NDCG@k: Normalized Discounted Cumulative Gain. Penalizes relevant results appearing lower. Target: > 0.7 for well-balanced dataset.
• MRR: Mean Reciprocal Rank. Average of the reciprocal rank of the first relevant result. Target: > 0.6 for challenging queries.
• Precision@k: % of relevant results in top k. Target: > 0.8@10.
• Recall@k: % of total relevant documents retrieved in top k. Target depends on context (precision/recall trade-off).

Create a manual validation set (100-500 queries with relevance judgments) and use these tools for incremental weight tuning.

Common mistakes and how to avoid them

• Forgetting score normalization: BM25 scores often range 0-100+, embedding cosine is 0-1. Normalize before combining.
• Fixed weights for all use cases: Always evaluate weights on real data. 0.5/0.5 doesn't work everywhere.
• Ignoring computational cost: Embeddings scale poorly with large corpora. Size your vector store correctly.
• Not re-indexing after model updates: If you change embedding models, recreate indices. Embeddings from different models aren't compatible.
• Assuming hybrid search is always better: For small datasets (< 10k docs) or very specific ones, pure BM25 may suffice. Always measure.

FAQ on hybrid search

Q: Do I need to index both BM25 and embeddings?
A: Yes, for true hybrid search. BM25 requires full-text indexed text, embeddings require pre-calculated vectors. Most modern vector databases natively support both.

Q: What embedding dimension should I choose?
A: 768-dim is standard (all-MiniLM-L6-v2). For very large datasets (> 10M docs), consider 384-dim. For maximum precision, 1024-dim. Trade-off: larger = more accurate but more storage.

Q: What if the query is very long (entire paragraph)?
A: Some embedding models have token limits (512 for many Sentence Transformers). Chunk the text, embed chunks, average resulting vectors. Or use recent models supporting longer contexts (e.g., BGE-M3).

Q: Is a single embedding better or multi-embedding (chunking)?
A: Multi-embedding (one embedding per text chunk) allows finer-grained retrieval and is often more accurate. Adds indexing complexity, but retrieval stays fast.

Q: Can I use BM25 + embeddings without a specialized vector database?
A: Yes, but slow. PostgreSQL + pgvector + tsquery (full-text) is a DIY option. For production, use RediSearch, Elasticsearch, Milvus, or Qdrant.

Conclusions

Hybrid BM25 + dense embeddings search is now the norm in modern information retrieval systems. It combines lexical precision and semantics in a single robust framework. Implementing it requires:

• A vector database supporting fusion (RediSearch, Elasticsearch, Milvus)
• An embedding model (Sentence Transformers, Ollama)
• A validation set for weight tuning
• Attention to latency, storage, and cost trade-offs

If you're building a RAG system, knowledge base, or modern search engine, hybrid search should be in your toolkit from day one. In the next article of this series, we'll explore advanced re-ranking strategies and fine-tuning embedding models for specific use cases.