HNSW in Redis: recall and latency measured on our vectors

In short. A Redis HNSW index does not always return the ten true neighbours: it almost always returns ten, and that "almost" can be measured. On the site's 2,426 vectors, with 300 queries that were never indexed and a FLAT index as the exact reference, the index created the way the ml-service does it (no parameter beyond type, dimension and distance) has a mean recall@10 of 0.9443: in 94 queries out of 300 at least one neighbour is missing, and in the worst case six out of ten are missing. Raising EF_RUNTIME from 10 to 50 takes the recall to 0.9963, to 200 it reaches 0.9997. The price, at this scale, is small: 0.2-0.3 ms per query. But the gain is small too, because FLAT costs 0.26-0.30 ms and is exact. With just over two thousand vectors, the honest question is not "which EF_RUNTIME", it is "do I really need an approximate index".

Method. Everything was run on 12 November 2026 on Redis Stack 7.4.7 in a container, with a Python client on loopback on a shared machine. The vectors were only read from the site's development instance (port 6379); the indexes live on another instance, which I switched off at the end of the work. I did not verify the production configuration, and I do not infer it.

What is measured, and against what

Recall in one sentence

An approximate index answers "give me the ten vectors closest to this one" by looking at only part of the graph. The recall@10 is the fraction of the ten true neighbours that appear in its answer: 1.0 means it found them all, 0.6 that it found six. It does not say whether the results are good for the user, it says how far the index departs from what a full scan would have answered. It is a measure of fidelity, and like all measures of fidelity it needs an original to compare with.

Why FLAT is the yardstick

The FT.CREATE page says to choose FLAT "when you have small datasets (less than 1 million vectors) or when perfect search accuracy is more important than search latency". A FLAT index compares the query with every vector and therefore makes no mistakes: it is the ideal reference for knowing what the HNSW should have answered. So, on the same vectors, I build a FLAT index and one or more HNSW ones, query them all with the same queries, and count how many of the FLAT's ten ids I find in the others. If the reference were itself approximate, the measurement would say nothing.

The environment, before the numbers

The versions, and why a container

The native Redis on my machine does not have the search module, so it cannot create a vector index. You can see it in two commands, and it explains why the ml-service logs "RediSearch module not loaded" and carries on without indexing when it runs against a Redis like this:

$ redis-server --version
Redis server v=8.0.5 sha=00000000:0 malloc=jemalloc-5.3.0 bits=64 build=d47fc40702b1fad8
$ redis-cli module list | tr '\n' ' '
name vectorset ver 1 path  args
$ redis-cli ft.create x on hash schema a text
ERR unknown command 'ft.create', with args beginning with: 'x' 'on' 'hash' 'schema' 'a' 'text'

The vectorset module that appears in the list is something else, a new data type with its own commands: it is not RediSearch and does not answer to FT.*. To do the measurement I started the redis/redis-stack-server image, which carries the search module, bound to 127.0.0.1 and on a port that is not the development one:

$ docker run -d --rm --name serie-redis-a10 -p 127.0.0.1:6398:6379 redis/redis-stack-server:latest
$ redis-cli -p 6398 info server | grep redis_version
redis_version:7.4.7
$ redis-cli -p 6398 module list | tr '\n' ' ' | grep -o 'name search ver [0-9]*'
name search ver 21020

The --rm makes the container disappear when I stop it and the 127.0.0.1 keeps the port from being reachable from outside. Two different versions, native 8.0.5 and 7.4.7 in Stack, are already a warning: the numbers that follow hold for 7.4.7 and for that version of the module, and I do not transfer them to others without redoing the test.

The site's vectors, read without touching them

The vectors are the articles' embeddings, one article:* hash per article and per language with an embedding field of 384 float32. Before using them I check two things: how many there are and that they are all of the expected size, because a single crooked vector makes an index fail. The command is read-only:

$ redis-cli -p 6379 --scan --pattern 'article:*' --count 1000 | wc -l
2426
$ redis-cli -p 6379 hstrlen article:1416:en embedding
1536
$ redis-cli -p 6379 hkeys article:1416:en | tr '\n' ' '
title article_id embedding slug excerpt lang

1,536 bytes are 384 numbers of 4 bytes: confirmation that the format is the one declared in the schema. In my script I then checked the length on all 2,426 keys and none was discarded. I used --scan and not KEYS because the first walks in blocks without stopping the server, while the second blocks it until it has finished: on an instance that others are using, it matters.

Building the indexes

The site's schema, line by line

The ml-service creates the articles index in create_index, in the file ml-service/app/storage/redis_vectors.py. For the vector field it passes VECTOR HNSW 6 TYPE FLOAT32 DIM 384 DISTANCE_METRIC COSINE: that 6 is the number of arguments that follow, three name-value pairs. There is no M, there is no EF_CONSTRUCTION, and the query at line 464 is KNN {k} @embedding $vec AS score, without EF_RUNTIME. So everything that matters holds for the defaults. I reproduce the same schema and ask Redis which values it chose:

$ redis-cli -p 6398 ft.create idx:articles on hash prefix 1 article: schema title text slug tag lang tag excerpt text article_id numeric embedding vector hnsw 6 type float32 dim 384 distance_metric cosine
OK
$ redis-cli -p 6398 ft.info idx:articles | grep -A14 '^embedding$' | tail -n +4 | head -14 | tr '\n' ' '
type VECTOR algorithm HNSW data_type FLOAT32 dim 384 distance_metric COSINE M 16 ef_construction 200

The FT.INFO of 7.4.7 shows M 16 and ef_construction 200 and does not show ef_runtime. For the search parameter the vectors page applies, which gives 10 as the default value; the same page reports 16 and 200 for the other two, and they match what the server declared. A fact follows that I did not expect: the site's datasets index, in semantic_search_datasets.py, sets M 16 and EF_CONSTRUCTION 200, which are the same values it would have got without writing them. It is an explicit documentation of the default, not a tuning.

Keeping the queries out

The risk of this measurement is self-reference: if I query the index with a vector it already contains, the first neighbour is the vector itself, in any algorithm, and the recall looks better than it is. To avoid it I pick 300 vectors at random and do not put them in the index: I use them only as queries. The other 2,126 end up in three indexes on different prefixes, one FLAT and two HNSW:

random.seed(7)
held = set(random.sample(names, 300))          # queries: never indexed
corpus = [n for n in names if n not in held]   # 2126 indexed vectors

mk('idx:flat',  'f:', 'FLAT', [])
mk('idx:hnsw',  'h:', 'HNSW', [])                                  # like the ml-service
mk('idx:hnsw4', 'g:', 'HNSW', ['M', '4', 'EF_CONSTRUCTION', '20']) # poor graph, for contrast

def q(idx, blob, ef=None, k=10):
    attr = '' if ef is None else '=>{$EF_RUNTIME: %d}' % ef
    return dst.execute_command('FT.SEARCH', idx,
        f'*=>[KNN {k} @embedding $B AS d]' + attr,
        'PARAMS', '2', 'B', blob, 'SORTBY', 'd', 'NOCONTENT', 'DIALECT', '2', 'LIMIT', '0', str(k))

The mk function composes the FT.CREATE command as above. The query parameter is passed with the attribute syntax, =>{$EF_RUNTIME: 50}, after the KNN part; NOCONTENT returns only the ids, because I need the ids and the latency. The poor graph (M 4, EF_CONSTRUCTION 20) is not a proposal: it serves to show that accuracy depends also on how the index was built, not only on how you query it.

The measurement

The recall calculation

The truth comes from the FLAT index: for each of the 300 queries I save the ten ids it returns. Then, for each configuration, the recall of a query is the size of the intersection between its ten ids and those ten, divided by ten. I repeat each configuration for three rounds and time with time.perf_counter around the call alone, so the times include the local network and the transfer of the reply:

truth = {n: ids(q('idx:flat', vecs[n])) for n in hq}
for name, idx, ef in cfg:
    for rnd in range(3):
        for n in hq:
            t = time.perf_counter(); r = q(idx, vecs[n], ef)
            lat.append((time.perf_counter() - t) * 1000)
            if rnd == 0:
                rec.append(len(set(ids(r)) & set(truth[n])) / 10)

I compute the recall only in the first round because the replies, with the index at rest, are the same; the following rounds serve for the latency. The number "how many queries below 1.0" is the one that tells a reader the most: a mean of 0.94 hides whether all lose a little or a few lose a lot, and the minimum says how badly a single search can go.

Why does Redis's HNSW with the defaults lose neighbours?

This is the full output, seed 7, with the indexes built the way the ml-service builds its own. The columns are the mean recall, the minimum recall, the number of queries below 1.0, the latency p50 over the three rounds and the 95th percentile:

config                     recall    min   <1.0 | p50 ms per round (3 rounds) | p95
FLAT                       1.0000    1.0      0 | 0.268 0.280 0.299 | 0.34
HNSW default               0.9443    0.4     94 | 0.193 0.165 0.158 | 0.24
HNSW EF_RUNTIME 10         0.9443    0.4     94 | 0.150 0.180 0.215 | 0.24
HNSW EF_RUNTIME 50         0.9963    0.9     11 | 0.270 0.269 0.244 | 0.35
HNSW EF_RUNTIME 200        0.9997    0.9      1 | 0.333 0.325 0.317 | 0.40
HNSW M4 EFC20 default      0.6643    0.0    237 | 0.167 0.143 0.163 | 0.19
HNSW M4 EFC20 EF 200       0.9747    0.4     54 | 0.320 0.285 0.322 | 0.41

Rows 2 and 3 are identical in recall, 0.9443 and the same 94 cases, because EF_RUNTIME 10 is the default: the latency difference between them (0.158-0.193 against 0.150-0.215) is the machine's noise, and it gives the measure of how much a comparison between times this small is worth. The figure that weighs is the min column: 0.4 means there is a search in which the index returned four true neighbours out of ten.

Why the first estimate was too optimistic

If I repeat the same test querying with 300 vectors that are already indexed, the result changes. The vector itself is always the first neighbour and the rest of the rows hold up a little better:

SEED 7 SELF
HNSW default               0.9837    0.6     41 | 0.176 0.172 0.176 | 0.23
HNSW EF_RUNTIME 50         0.9997    0.9      1 | 0.233 0.224 0.232 | 0.29
HNSW EF_RUNTIME 200        1.0000    1.0      0 | 0.335 0.347 0.321 | 0.41

From 0.9837 to 0.9443 is the price of self-reference, and it is worth almost four points out of a hundred: whoever measures the recall by querying the index with vectors it contains gets a prettier number than the one a user will see. To rule out the doubt that the seed is lucky I repeated the measurement with two others, 11 and 23: in the three cases the default recall is 0.9443, 0.9390 and 0.9487, with 94, 100 and 89 queries out of 300 below 1.0. The spread is under one point: the number is stable for this set of vectors.

EF_RUNTIME: the syntax and the traps

The parameter can be passed per query, and this is the right way to try it: it does not change the index, it does not require a rebuild, and it can be compared with another value on the same instance. But the syntax is strict, and Redis's errors say little. Three cases I met, with the exact messages:

$ redis-cli -p 6398 ft.search idx:f '*=>[KNN 3 @v $B]=>{$EF_RUNTIME: 50}' params 2 B abcdabcdabcdabcd dialect 2
Error parsing vector similarity parameters: Invalid option
$ redis-cli -p 6398 ft.search idx:h '*=>[KNN 3 @v $B]' params 2 B abcdabcd dialect 2
Error parsing vector similarity query: query vector blob size (8) does not match index's expected size (16).
$ redis-cli -p 6398 ft.search idx:h '*=>[KNN 3 @v $B]' params 2 B abcdabcdabcdabcd
Syntax error at offset 1 near >[

The first is EF_RUNTIME on a FLAT index: the page explains it ("for FLAT there are no runtime parameters") and the server answers Invalid option without saying which. The second is a vector of the wrong size, the most frequent error when you change embedding model. The third is forgetting DIALECT 2, without which the KNN syntax cannot be read. On an empty index the first case gives no error, but returns zero results: an experiment run before loading the data would have hidden the problem.

Which EF_RUNTIME is worth using?

Over the three seeds, with the queries kept out, the three knobs read like this:

• EF_RUNTIME 10 (default): recall 0.9443 / 0.9390 / 0.9487, between 89 and 100 queries out of 300 with at least one missing neighbour, p50 from 0.148 to 0.193 ms.
• EF_RUNTIME 50: recall 0.9963 / 0.9970 / 0.9987, between 2 and 11 imperfect queries, p50 from 0.211 to 0.273 ms.
• EF_RUNTIME 200: recall 0.9997 / 1.0000 / 1.0000, from 0 to 1 imperfect query, p50 from 0.311 to 0.384 ms.
• FLAT: recall 1.0000 by definition, p50 from 0.263 to 0.299 ms.

The reading that matters is between the lines: going from 10 to 50 buys almost all the recall there is to buy (from 0.94 to 0.996) and costs between 0.04 and 0.10 ms; going from 50 to 200 buys at most a few thousandths and costs another 0.1 ms. With EF_RUNTIME 200 the approximate index becomes slower than the exact one, because at two thousand vectors the search visits almost the whole graph and on top of that pays for the navigation. There is no right value for everybody: there is a point beyond which you buy nothing more, and here it is close to 50.

What happens with a poor graph

The M4 EFC20 row is there to remind us that accuracy does not depend only on the search parameter. With a graph built with few links per node (M 4) and little care (EF_CONSTRUCTION 20) the default recall drops to 0.6643, with 237 queries out of 300 imperfect and a minimum of 0, that is a search that found none of the ten true neighbours. Raising EF_RUNTIME to 200 takes it to 0.9747: it does not bring it back to 1 because the graph, once badly built, cannot be repaired from outside. That is why HNSW indexes have two knobs, one you pay for when you build and one you pay for at every search.

The cost: time and memory

The latency, and how much the noise is worth

The times must be read with suspicion. They are medians over 300 calls per round, a single client, loopback, on a machine where other things were running, and it shows: the two identical rows (default and EF_RUNTIME 10) differ by 0.05 ms in some rounds. A comparison between two configurations makes sense only when the difference exceeds that noise: the default against EF_RUNTIME 50 exceeds it, EF_RUNTIME 10 against the default does not. That HNSW's overall gain over FLAT, at this scale, is about 0.1 ms is a true and small result.

Memory: HNSW weighs more

The other cost is space, and FT.INFO reports it per index, in the vector_index_sz_mb field:

idx:flat vector_index_sz_mb b'4.613433837890625'
idx:hnsw vector_index_sz_mb b'5.4040069580078125'

The graph adds about 0.79 MB to an index of 4.61, that is 17%, for 2,126 vectors. It is a negligible amount here and proportional to the number of links per node, so it grows with M: the vectors page says a higher value also increases memory. A caveat: 4.61 MB for 2,126 vectors of 1,536 bytes is more than the 3.11 MiB of raw data I expected, and I cannot explain where the difference comes from. I report the field as I read it, and I do not use it to size anything.

With a few thousand vectors, do you really need an HNSW index?

At 2,126 vectors I would choose FLAT. It gains zero ambiguity, because the result is always the exact one, and loses 0.1 ms on a page that spends many more elsewhere: the approximation buys nothing the user can notice. HNSW starts to pay off when full scans become the dominant cost, and I did not measure that point: I do not know at how many vectors it falls, and that is why I do not write it. If the site were to stay on HNSW, I would set EF_RUNTIME to 50 in the query and not in the index, because the query changes at every release and the index gets rebuilt.

• Measure the recall on your own vectors, not on those of a tutorial: it depends on the distribution of the data and changes from one corpus to another.
• Query with vectors outside the index: those already inside hand you the first neighbour and inflate the number.
• Compare with a FLAT on the same set before choosing the algorithm, and report the minimum as well as the mean.
• Try EF_RUNTIME per query, starting from 50: it costs one line and does not change the index.
• Write the Redis version next to every number: a different module may behave differently.

The five entries are in order of execution, and the first three are the ones you do not skip: without an exact reference and without external queries, the number you get cannot be interpreted. The last two are maintenance. The step that cost the most, in my case, was the second, and it was not hard: all it took was setting aside three hundred vectors before creating the indexes.

Where the measurement does not reach

This test has limits that are better declared than discovered later:

• The queries are articles, not questions: the site's user writes short sentences, whose embedding may fall in different zones of the space. I had no embedding model on the machine to produce them, and I did not measure that case.
• The site's searches have filters (by language, for example): the FT.SEARCH page describes a mode in which, when the documents that pass the filter are few, the server computes the distance of each and returns exact results even on an approximate index. I did not measure the filtered queries.
• The instance is a development one: the vectors are the same as the site's, but I did not verify the production version and configuration.
• Single client, loopback: no concurrency. The times say how much one search costs, not how much the service sustains under load.

None of these limits overturns the conclusion, but each narrows it. The first is the most important, because the recall depends on how the questions are made as well as on the documents; to measure it you would need a sample of real questions and their embedding, and it is the step I would take before changing any parameter.

What you can do now

You can build an exact reference with a FLAT index, keep three hundred vectors out as true queries and count how many neighbours an HNSW index loses, with the mean, the minimum and the number of imperfect searches. You know that Redis's defaults (M 16, EF_CONSTRUCTION 200, EF_RUNTIME 10) give, on the site's vectors, a recall of about 0.94, that EF_RUNTIME 50 takes it above 0.99 and that going to 200 at this scale costs more than the exact index. And you know that before choosing the algorithm it is worth asking whether you need an approximate one.

To know how much space each vector takes in the hash that holds it there is the article on data structures and what they cost; for the instance's overall memory, the one on maxmemory, eviction and OOM; for the method of measuring latency and memory, measuring Redis. The context in which the site uses these vectors, that is RAG, is in the article on the complete RAG setup with RediSearch and in the one on reranking with a cross-encoder.

Have a vector index and not sure how wrong it is? Book a call →

Sources

• Redis, Vectors - the HNSW parameters with their default values (M 16, EF_CONSTRUCTION 200, EF_RUNTIME 10), the warning that increasing M increases memory, the per-query EF_RUNTIME syntax and the fact that for FLAT there are no runtime parameters. Consulted on 2026-11-12.
• Redis, FT.CREATE - the sentence on choosing FLAT for sets under a million vectors or when perfect accuracy matters more than latency. Consulted on 2026-11-12.
• Redis, FT.SEARCH - the brute-force mode when the filter leaves few documents, with exact results even on an approximate index. Consulted on 2026-11-12.
• Malkov and Yashunin, Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs - the algorithm from which HNSW comes; I checked title and authors on the arXiv page, I did not reread the content. Consulted on 2026-11-12.
• Own measurements: the script and the commands in the blocks, run on 12 November 2026 on Redis Stack 7.4.7 in a container (redis/redis-stack-server:latest), with the vectors read from port 6379.
• This site's code: ml-service/app/storage/redis_vectors.py (create_index, the query at line 464) and ml-service/app/core/semantic_search_datasets.py (lines 33-35), read on 12 November 2026.

The first four entries are documentation and a publication: they serve to check that I read the parameters and their defaults correctly, and if a page changes the numbers, the FT.INFO command on your version has the last word. The last two can be redone: the script, with another seed, gives a measurement that differs in the details and agrees in substance.