Which AI model? Llama 3.1 vs Mistral vs Qwen vs Gemma deep-dive 2026

Choosing a model family with what can be verified

Comparing Llama, Mistral, Qwen and Gemma with a ranking of scores is the quickest way to choose badly: public scores measure general tasks, mostly in English, and say nothing about your documents. This article compares the four families on what can be read in a configuration or in a license and can be calculated: the architecture, the cache memory, the context that really fits in a machine, the license, the support for tools. Quality on your task remains a measurement to be made, with the method of the article on choosing the size.

The result, anticipated: for a small or medium company with a 16 GB machine, the most concrete difference among the four families is not the ranking but the usable context. At equal size, one family may let you give the model a whole contract, another only a chapter, because of an architecture fact that is calculated in three lines.

What the configurations say

Every model publishes a configuration file with the dimensions of its architecture. Four numbers are enough to estimate the cache memory: the number of layers, the number of attention heads for keys and values (KV heads), the size of each head and the maximum context. What follows comes from the configurations read on 2026-11-13.

• Llama 3.1 8B: 32 layers, 32 attention heads and 8 KV heads, head size 128, context up to 131,072 tokens.
• Mistral 7B v0.3: 32 layers, 32 heads and 8 KV heads, head size 128, maximum context 32,768 tokens.
• Qwen2.5 7B: 28 layers, 28 heads and only 4 KV heads, head size 128, maximum context 32,768 tokens.
• Gemma 2 9B: 42 layers, 16 heads and 8 KV heads, head size 256, maximum context 8,192 tokens, with a sliding window of 4096.

The ratio between attention heads and KV heads is the idea of grouped-query attention (GQA): several query heads share the same key and value head, and the cache, which stores precisely keys and values, shrinks in proportion. Llama and Mistral share each KV head among four heads, Qwen among seven, and for this reason Qwen has the smallest cache per token; Gemma has a double-size head and more layers, so the largest. It is a design difference that can be calculated, not a judgment of quality.

Cache per token and the context that fits in memory

The cache per token is calculated by multiplying two matrices (keys and values) by layers, KV heads, head size and two bytes per 16-bit floating point value. The program below does it for the four families, and then answers the practical question: given a server with 16 or 32 GB of RAM, after removing the weights of the file (from the Ollama library page), the overhead and the reserve for the system, how many tokens of context fit for each request, with one, two or four requests in parallel.

The assumptions are three constants at the top of the code, to be replaced with your own measurements: the cache at 16 bits, one gibibyte of overhead, one and a half gibibytes for the system. For Gemma the calculation is a ceiling, because the sliding window reduces the real cache and the program does not model it. The context is capped at the maximum declared by each model.

"""What context fits in a machine, for four model families.

Architecture: config.json of the models on Hugging Face (copies declared in the article).
File size and context window: Ollama library pages.
For Gemma 2 the cache is a CEILING: its sliding-window attention of 4096
tokens on part of the layers would reduce the real cache, and it is not modeled here.
ASSUMPTION: cache in f16, 1 GiB of overhead, 1.5 GiB for the system.
"""
GIB = 1024 ** 3
OVERHEAD = 1.0 * GIB
SYSTEM = 1.5 * GIB

# name: (layers, KV heads, head dim, declared maximum context, GB Ollama file)
M = {
    "llama3.1:8b": (32, 8, 128, 131072, 4.9),
    "mistral:7b": (32, 8, 128, 32768, 4.4),
    "qwen2.5:7b": (28, 4, 128, 32768, 4.7),
    "gemma2:9b": (42, 8, 256, 8192, 5.4),
}


def kv_token(layers, kv, hd):
    return 2 * layers * kv * hd * 2


print(f"{'model':13s} {'KiB/token':>9s} {'ctx max':>8s} {'cache at maximum':>17s}")
for n, (st, kv, hd, cmax, gb) in M.items():
    b = kv_token(st, kv, hd)
    print(f"{n:13s} {b // 1024:9d} {cmax:8d} {b * cmax / GIB:14.1f} GiB")


def max_context(ram_gb, n, parallel):
    st, kv, hd, cmax, gb = M[n]
    free = ram_gb * 1e9 - SYSTEM - OVERHEAD - gb * 1e9
    if free <= 0:
        return 0
    return min(cmax, int(free / (kv_token(st, kv, hd) * parallel)))


for ram in (16, 32):
    print(f"\nmaximum context that fits in {ram} GB of RAM (tokens, per request)")
    for par in (1, 2, 4):
        cells = "  ".join(f"{n}: {max_context(ram, n, par):6d}" for n in M)
        print(f"  {par} in parallel -> {cells}")

# Output:
# model         KiB/token  ctx max  cache at maximum
# llama3.1:8b         128   131072           16.0 GiB
# mistral:7b          128    32768            4.0 GiB
# qwen2.5:7b           56    32768            1.8 GiB
# gemma2:9b           336     8192            2.6 GiB
#
# maximum context that fits in 16 GB of RAM (tokens, per request)
#   1 in parallel -> llama3.1:8b:  64206  mistral:7b:  32768  qwen2.5:7b:  32768  gemma2:9b:   8192
#   2 in parallel -> llama3.1:8b:  32103  mistral:7b:  32768  qwen2.5:7b:  32768  gemma2:9b:   8192
#   4 in parallel -> llama3.1:8b:  16051  mistral:7b:  17005  qwen2.5:7b:  32768  gemma2:9b:   5751
#
# maximum context that fits in 32 GB of RAM (tokens, per request)
#   1 in parallel -> llama3.1:8b: 131072  mistral:7b:  32768  qwen2.5:7b:  32768  gemma2:9b:   8192
#   2 in parallel -> llama3.1:8b:  93138  mistral:7b:  32768  qwen2.5:7b:  32768  gemma2:9b:   8192
#   4 in parallel -> llama3.1:8b:  46569  mistral:7b:  32768  qwen2.5:7b:  32768  gemma2:9b:   8192

Three facts can be read. The first: the cache of Llama 3.1 at the maximum context of 131,072 tokens is worth 16 GiB, more than all the RAM of a 16 GB machine: the "128K" context of the catalog is not usable on the small machine, and with one request at a time about 64,000 tokens fit. The second: Qwen stays at 32,768 tokens even with four requests in parallel, because its cache is the smallest. The third: with four requests in parallel on 16 GB, Mistral drops to about 17,000 tokens and Llama to about 16,000; parallelism is paid for in context.

The context that fits is not the context worth using: the longer the text, the longer the model takes to read it before answering, and reading the prompt is the slowest part on a CPU. For a document assistant, where the index chooses the relevant passages (as described in the article on hybrid retrieval), a few thousand tokens are usually enough, and the maximum context matters less than the cache per request. If instead you have to give the model a whole document, the context is the constraint to start from. Ollama's documentation indicates that the default context is 4k tokens below 24 GiB of VRAM; the page speaks of VRAM and I have not verified the value for CPU-only execution, so set the OLLAMA_CONTEXT_LENGTH variable explicitly.

Licenses: the line that changes with the size

The license is read from the model card, size by size, because it can change inside the same family. From the Hugging Face cards of 2026-11-13: Mistral 7B Instruct v0.3 is Apache-2.0; Qwen2.5 7B and 14B are Apache-2.0, while Qwen2.5 3B has a research license that requires asking for a license for commercial use; Llama 3.1 has Meta's own license, with the threshold of 700 million monthly active users, the obligation to display "Built with Llama" and the reference to the acceptable use policy; Gemma 2 has Google's own license, which I have not read in full here.

The consequence is a filter to apply first: if your company cannot accept a license with conditions (for example because you distribute the model inside a product), the only clean choices are those with Apache-2.0. For internal use the matter is less pressing, but reading the text remains necessary: a license is not deduced from the name of the family.

Tools and Italian: what cannot be said from here

Two typical questions, "does it work well in Italian?" and "can it call functions?", do not have an answer verifiable from primary sources within the space of an article. On the second there is a partial datum: in the Ollama library the llama3.1, mistral and qwen2.5 pages carry the "tools" tag, while the gemma2 one does not. It is an indication that the function-calling format is supported by the model in that library, not a measure of how reliable it is: a model can have the tag and get an argument wrong once in ten.

For Italian and for the reliability of tools you need a test on your own case. Prepare twenty real questions with the expected answer, run them on the candidates left after the license and memory filters, and count the errors; remember from the other article that with few questions a result is too uncertain to declare a winner, and that small differences between similar models are almost impossible to demonstrate. If two models are tied on your set, the one with the simpler license and the smaller cache wins.

A four-step selection procedure

The order of the steps is not random: each one eliminates candidates with a check that is cheaper than the next.

1. License: discard the models whose conditions you cannot accept.
2. Memory and context: with the program above, discard those that do not give the context you need on the machine you have.
3. Capability on your task: test the remaining ones on your set of questions, with the functionality you need (tools, language).
4. Cost and speed: among equals, take the smallest; the cost comparison is in the article on costs, and speed depends on memory bandwidth as the piece on the server shows.

The advantage of the order is that the first two steps require no execution and cost no time. The test of the third step, which costs time, is done on few models. And the fourth matters only if the third has not already designated a winner: a larger model that makes fewer mistakes is often worth the cost difference, and it is the cost that says so, not the ranking; to understand what happens under load there is the article on scalability (from 1 to 50 users).

Frequently asked questions

What you can do now

You can read from a configuration the four numbers that fix the cache, you can calculate how much context fits in your machine and how it falls with parallelism, and you know that the license changes with the size inside the same family. You have a procedure that applies the cheap checks first, and you know that Italian and tool reliability remain to be measured on your own case. The next step of the series is the evaluation of the quality of retrieval and answers.

The same series also includes privacy and the GDPR with an in-house model and the recap with the decision sheet.