08 - On-prem inference with Ollama, vLLM, llama.cpp

You want to run an open model on your own machine and you need three answers before downloading anything: does it fit in memory, how many requests does it handle together, and which program is worth using to serve it. Memory is estimated with two items, the weights (parameters times bits per weight) and the key-value cache, which grows with the length of the sequences and with the number of active requests; the ability to serve more requests depends on how the engine manages that cache and on how it builds groups of requests; the program you choose by looking at what its documentation declares it optimises and at what your hardware allows. None of the three answers requires a model in order to be understood: calculations and small simulations are enough, and you find them below.

This is the eighth and last article of the series Modern Artificial Intelligence. You arrive from the one on reasoning models, which explains why more compute per answer has a cost, and this article shows where that cost is paid in memory and in time. Basic Python with numpy is enough. No model is downloaded or called, and for this reason you will not find measurements of tokens per second: they depend on your machine, and a number taken from elsewhere would say nothing about it.

How much the weights weigh, and what is lost by quantising them

A model of 7 billion parameters in 16-bit precision occupies, for the weights alone, 7e9 times 2 bytes, that is about 13 GiB. Quantising means representing each weight with fewer bits: at 8 bits the memory halves, at 4 bits it drops to a quarter, plus a small share for the scales needed to reconstruct the values. The cost is a rounding error, and it is worth seeing how big it is and what it depends on, before trusting a percentage read somewhere.

The listing quantises in groups of 64 weights, with a symmetric scheme and rounding to the nearest: for each group a scale is computed from the largest weight in absolute value, the weights are divided by the scale, rounded to integers and reconstructed. I apply it to a synthetic 512 by 512 matrix of Gaussian weights and then to the same matrix in which 1% of the weights are 30 times larger, because in real models some weights and some activations have outlier values.

"""Group-wise weight quantization: memory and error, on synthetic weights.
Scheme: symmetric, round-to-nearest, one fp16 scale per group of 64 weights."""
import numpy as np

GROUP = 64


def quantize(w, bit):
    """Returns the weights reconstructed after group-wise quantization."""
    qmax = 2 ** (bit - 1) - 1
    g = w.reshape(-1, GROUP)
    scale = np.abs(g).max(axis=1, keepdims=True) / qmax
    scale = np.where(scale == 0, 1.0, scale)
    q = np.clip(np.round(g / scale), -qmax, qmax)
    return (q * scale).reshape(w.shape), scale


def measure(name, w, x, normal):
    print(name)
    for bit in (8, 4):
        wq, scale = quantize(w, bit)
        err = wq - w
        mse = float((err ** 2).mean())
        predicted = float((scale ** 2 / 12).mean())     # second calculation: delta^2 / 12
        rel_weights = np.sqrt((err[normal] ** 2).mean()) / w[normal].std()
        rel_output = np.linalg.norm(x @ wq - x @ w) / np.linalg.norm(x @ w)
        print(f"  {bit} bit: relative error on normal weights {rel_weights:.4f}, on the output {rel_output:.4f}, "
              f"MSE measured/predicted {mse / predicted:.2f}")


rng = np.random.default_rng(3)
W = rng.normal(0.0, 0.02, size=(512, 512))
X = rng.normal(size=(64, 512))
measure("Gaussian weights", W, X, np.ones(W.shape, dtype=bool))

W_out = W.copy()                                    # 1% of weights 30 times larger
idx = rng.choice(W_out.size, size=W_out.size // 100, replace=False)
W_out.flat[idx] *= 30.0
normal = np.ones(W.shape, dtype=bool)
normal.flat[idx] = False
measure("1% of weights 30 times larger", W_out, X, normal)

print("memory of a 7-billion-parameter model (weights only):")
N = 7e9
for name, bits_per_weight in [("fp16", 16.0), ("int8 + scales", 8 + 16 / GROUP),
                              ("4 bit + scales", 4 + 16 / GROUP)]:
    print(f"  {name:14s} {bits_per_weight:5.2f} bit/weight -> {N * bits_per_weight / 8 / 2**30:5.1f} GiB")

# Output:
# Gaussian weights
#   8 bit: relative error on normal weights 0.0059, on the output 0.0060, MSE measured/predicted 0.99
#   4 bit: relative error on normal weights 0.1074, on the output 0.1079, MSE measured/predicted 0.98
# 1% of weights 30 times larger
#   8 bit: relative error on normal weights 0.0504, on the output 0.0162, MSE measured/predicted 0.98
#   4 bit: relative error on normal weights 0.5419, on the output 0.1751, MSE measured/predicted 0.35
# memory of a 7-billion-parameter model (weights only):
#   fp16           16.00 bit/weight ->  13.0 GiB
#   int8 + scales   8.25 bit/weight ->   6.7 GiB
#   4 bit + scales  4.25 bit/weight ->   3.5 GiB

In the code, quantize computes scale as the maximum of the group divided by qmax, and this line decides everything: a single large weight widens the quantisation step of all 63 neighbours. As a second, independent calculation I compare the measured squared error with the theoretical prediction for uniform rounding, step squared divided by 12: in the Gaussian case the ratio is 0.99 and 0.98, so the theory holds. In the case with outliers at 4 bits the ratio falls to 0.35, because the step becomes larger than the ordinary weights, which end up almost all rounded to zero: the error on them is their value, no longer a uniform error, and the formula does not hold.

The row that matters is the one on the normal weights, that is the 99% that are not outliers. Without outliers the relative error is 0.006 at 8 bits and 0.107 at 4 bits; with 1% of outliers it rises to 0.050 and 0.542: the ordinary weights are quantised worse because they share the group with a bulky companion, and almost half of the groups (1 minus 0.99 to the power 64, about 47%) contain at least one. The error on the output is more contained because in this synthetic case the large weights, reconstructed with a small relative error, weigh a lot in the norm of the output. This is the problem the methods in the literature tackle: GPTQ quantises in a single pass using approximate second-order information, and AWQ observes that not all weights have the same importance and that protecting 1% of them reduces the error a lot, identifying them from the distribution of the activations and not from that of the weights. My listing does not implement them: it shows why they are needed.

The key-value cache: how much a token weighs, how much is wasted

During generation, each layer keeps the keys and values of the tokens already seen so as not to recompute them. Its weight per token is 2 (keys and values) times the number of layers, times the key-value heads, times the head dimension, times the bytes per element. The listing uses an invented model configuration, which corresponds to no product, and checks it by creating an array with those dimensions and reading its nbytes.

"""KV cache: how much a token weighs, and how much space a fixed reservation
wastes compared with block allocation. INVENTED model configuration, only for the arithmetic."""
import math
import numpy as np

LAYERS, KV_HEADS, HEAD_DIM = 32, 8, 128      # example model, not a real product
BYTES = 2                                     # fp16

per_token = 2 * LAYERS * KV_HEADS * HEAD_DIM * BYTES      # 2 = keys + values
check = np.empty((2, LAYERS, KV_HEADS, HEAD_DIM), dtype=np.float16).nbytes
assert per_token == check
print(f"KV cache per token: {per_token} bytes = {per_token // 1024} KiB")
print(f"a sequence of 4096 tokens: {per_token * 4096 / 2**20:.0f} MiB")

# synthetic requests: total length (prompt + answer) lognormal, clipped at 2048
rng = np.random.default_rng(11)
lengths = np.clip(rng.lognormal(mean=5.5, sigma=0.8, size=1000), 16, 2048).astype(int)
MAX_LEN, BLOCK, POOL = 2048, 16, 120_000      # POOL: token slots available in memory

used = int(lengths.sum())
fixed = MAX_LEN * len(lengths)
in_blocks = sum(math.ceil(n / BLOCK) * BLOCK for n in lengths)
print(f"mean length {lengths.mean():.0f} tokens, median {int(np.median(lengths))}")
print(f"fixed reservation at {MAX_LEN}: utilisation {used / fixed:.1%}")
print(f"blocks of {BLOCK}: utilisation {used / in_blocks:.1%} "
      f"(mean waste {(in_blocks - used) / len(lengths):.1f} tokens per request, "
      f"expected about {(BLOCK - 1) / 2:.1f})")

def admitted(cost):
    """How many requests, in order, fit before the pool runs out."""
    total = n = 0
    for c in cost:
        if total + c > POOL:
            break
        total += c
        n += 1
    return n

fixed_cost = [MAX_LEN] * len(lengths)
block_cost = [math.ceil(n / BLOCK) * BLOCK for n in lengths]
print(f"requests in memory with {POOL} slots: fixed reservation {admitted(fixed_cost)}, "
      f"blocks {admitted(block_cost)}")

# Output:
# KV cache per token: 131072 bytes = 128 KiB
# a sequence of 4096 tokens: 512 MiB
# mean length 338 tokens, median 255
# fixed reservation at 2048: utilisation 16.5%
# blocks of 16: utilisation 97.8% (mean waste 7.6 tokens per request, expected about 7.5)
# requests in memory with 120000 slots: fixed reservation 58, blocks 350

With those numbers a token weighs 128 KiB and a sequence of 4096 tokens occupies 512 MiB: for this reason the available memory, more than the weights, limits how many requests you can handle together. The second part of the listing simulates 1000 requests of synthetic length (lognormal, mean 338 tokens, clipped at 2048). If for each one you reserve the space of the maximum length, the mean utilisation is 16.5%; if you assign the memory in blocks of 16 tokens, as they are needed, it is 97.8% and the waste is that of the last block, 7.6 tokens per request against the 7.5 expected. With a pool of 120,000 slots the fixed reservation admits 58, the blocks 350.

The numbers in this simulation are mine and the distribution is invented, but the logic is that of the PagedAttention work (Kwon and colleagues, 2023): the authors observe that in existing systems only 20.4%-38.2% of the cache memory contains actual token states, and propose to manage it like the virtual memory of an operating system, in blocks; in their work vLLM's default block is 16 tokens. They report for vLLM a near-zero waste and a throughput increase of 2-4 times at the same latency compared with the baseline systems, with a more marked improvement for longer sequences and larger models: these are their measurements, on their workloads, and I did not redo them.

Continuous batching: do not wait for the slowest request

The second waste is in time. An engine that serves fixed groups of requests starts with eight, and restarts only when the longest has finished: the other seven leave their place empty, idling, for the whole difference in duration. Continuous batching assigns a free place to the next request as soon as one becomes free. The listing simulates it in generation steps, with 200 requests of synthetic duration (mean 75 steps, maximum 400) and eight places.

"""Static batching against continuous batching, in generation steps.
Each step produces one token for every active request; synthetic durations, fixed seed."""
import math
import numpy as np

rng = np.random.default_rng(5)
DURATIONS = np.clip(rng.lognormal(mean=4.0, sigma=0.9, size=200), 4, 400).astype(int)
B = 8                                           # requests served together


def static(durations):
    """Groups of B: it restarts only when the longest request of the group finishes."""
    steps = 0
    for i in range(0, len(durations), B):
        steps += int(durations[i:i + B].max())
    return steps


def continuous(durations):
    """As soon as a request finishes, its slot goes to the next one in the queue."""
    queue = list(durations)
    active = [queue.pop(0) for _ in range(min(B, len(queue)))]
    steps = 0
    while active:
        steps += 1
        active = [d - 1 for d in active]
        active = [d for d in active if d > 0]
        while len(active) < B and queue:
            active.append(queue.pop(0))
    return steps


tot = int(DURATIONS.sum())
bound = math.ceil(tot / B)
ss, sc = static(DURATIONS), continuous(DURATIONS)
print(f"200 requests, mean duration {DURATIONS.mean():.0f} steps, maximum {DURATIONS.max()}, B = {B}")
print(f"lower bound (total tokens / B): {bound} steps")
print(f"static    : {ss} steps, slot occupancy {tot / (ss * B):.1%}")
print(f"continuous: {sc} steps, slot occupancy {tot / (sc * B):.1%}")
assert bound <= sc <= ss

# Output:
# 200 requests, mean duration 75 steps, maximum 400, B = 8
# lower bound (total tokens / B): 1874 steps
# static    : 5318 steps, slot occupancy 35.2%
# continuous: 2000 steps, slot occupancy 93.7%

The static function adds up, for each group of eight, the maximum duration; continuous replaces every finished request with the first in the queue. The lower bound, total tokens divided by eight, is 1874 steps and the final assertion checks that neither goes below it. Static batching takes 5318 steps, with an average slot occupancy of 35.2%; continuous batching 2000, with 93.7%. The simulation ignores the cost of reading the prompt, the memory limits of the previous section and the latency of the single request: it does not predict the real gain, it shows that the waste of the fixed group grows with the variance of the durations, and that it all lies in how the places are filled. The vLLM documentation lists among its features both PagedAttention and the continuous batching of incoming requests, and the PagedAttention paper recalls as earlier work the iteration-level scheduling of Orca.

Ollama, llama.cpp, vLLM: what they declare they do

I report to you what their official pages declare, consulted on 2026-11-13. I did not run any of the three, and I give no speed comparisons, because I did not measure them.

The table ranks nothing, and the choice comes from the measurements you made above. If you are a single person on a laptop, the weight memory and the quantisation matter more than batching, and the tools that focus on portability and quantisation are the natural candidates. If you serve many requests together on a graphics card, the cache and the filling of the places matter, which is precisely what vLLM declares it optimises. If you expose the engine on a network, put it behind a proxy with authentication and not on a public interface: I did not verify which of these tools offer authentication of their own, and I do not claim it.

Local does not mean compliant

Running the model on your own server avoids sending the prompts to an external provider for inference, and this changes who processes the data. It does not, however, remove the obligations of whoever processes them nor those of whoever puts an artificial intelligence system into service, and the texts change: Regulation (EU) 2026/1744 of 8 July 2026, which I read on EUR-Lex on 2026-11-13, amends Regulation (EU) 2024/1689 and moves, for example, the date of application of the obligations for the high-risk systems of Annex III to 2 December 2027. This is not legal advice: if your use touches personal data or falls within a regulated area, the text in force and a professional in the field are the right place.

What I did not measure

The three listings use synthetic weights, requests and durations, with fixed seeds, and an invented model configuration: they prove the mechanisms and the relationships, not the performance of a program or a machine. The PagedAttention figures (20.4%-38.2%, 2-4 times) belong to its authors. I did not measure tokens per second, latencies, the quality of a quantised model's answers or energy consumption, and I did not run the three tools: for this reason the table reports only what they declare. The features of software that changes continually must be re-verified on the project's page on the day you decide.

What you can do now

You can estimate whether a model fits, by adding quantised weights and cache; you can say why a single outlier value worsens the quantisation of its neighbours; you can explain why reserving the maximum length wastes memory and why the fixed group wastes time; and you can read the documentation of an inference engine asking yourself which of these wastes it declares it reduces.

Summary of the series

In eight stages, the series has covered the elements on which modern systems based on language models rest, from the architecture to how to run them. Here they all are, in order.

1. Transformer Architecture: attention, encoder, decoder
2. LLM scaling laws and emergent capabilities
3. Fundamental RAG: embeddings, vector store, retrieval
4. Advanced RAG: re-ranking, hybrid search, chunk strategy
5. AI Agents: ReAct, tool-use, planning
6. Model Context Protocol (MCP): integration standard
7. Reasoning models: o1, DeepSeek-R1, RLHF
8. On-prem inference with Ollama, vLLM, llama.cpp (this article)

The order makes sense: the first two articles explain what a model is and how it grows, the two on retrieval explain how to give it knowledge it does not have, the two on agents and protocol how to make it use tools, the one on reasoning how to spend more compute per answer, and this one where that compute runs and how much it costs in memory and in time. If you read in order, you saw the same method repeated: a mechanism built in small, a second confirming calculation, and the declaration of what the experiment does not say. It is the method worth taking away, more than any single figure, because figures change and the method stays valid. Go back to the index of the series to pick up a topic from where you need it.