Scaling Ollama: from 1 to 50 concurrent users on Hetzner VPS

Will your internal assistant hold up for fifty people? You work it out before you buy

The question "how many concurrent users does a server with Ollama sustain" has no answer in users. It has an answer in tokens per second: a model generates text at a speed that depends on the machine, every answer consumes a certain number of tokens, and the number of people you can serve follows from these two quantities and from how often each one asks for something. This article builds that calculation, applies it to a constructed scenario (fifty people querying a company assistant) and tells you where memory becomes the constraint before the processor does.

The result it is worth anticipating is uncomfortable: on a server without a GPU, fifty users asking for a long answer every two minutes require an aggregate speed that a single small server hardly offers. How to choose the model that produces it is in the article on Llama, Mistral, Qwen and Gemma; what it costs to keep it all running is in the analysis of 12 months of costs. Here we deal with sizing and with not letting the service collapse when the load exceeds capacity.

First you measure: tokens per second, from the answer itself

You need no external tool to know your server's speed: every /api/generate response from Ollama reports the number of tokens generated (eval_count), the time spent generating them in nanoseconds (eval_duration), the time spent reading the prompt (prompt_eval_duration) and the model loading time (load_duration), as the API documentation consulted on 2026-11-13 describes. Dividing the first two gives the tokens per second. The program below does the arithmetic on a sample JSON copied from the documentation: the example shows the calculation, not the speed of a real machine, and the value it produces must not be taken as a reference for your server.

"""Tokens per second from the fields Ollama returns in every /api/generate response."""
import json


def token_al_secondo(risposta):
    # eval_count: tokens generated; eval_duration: nanoseconds spent generating them
    return risposta["eval_count"] / (risposta["eval_duration"] / 1e9)


def tempo_prompt_s(risposta):
    # prompt_eval_duration: nanoseconds spent reading the prompt that is not cached
    return risposta["prompt_eval_duration"] / 1e9


# Sample JSON copied from the official API documentation: it serves to show the arithmetic,
# it does NOT describe the speed of any real machine.
esempio = json.loads("""{"total_duration": 174560334, "load_duration": 101397084,
 "prompt_eval_count": 11, "prompt_eval_duration": 13074791,
 "eval_count": 18, "eval_duration": 52479709}""")

print(f"generation: {token_al_secondo(esempio):.0f} tokens/s")
print(f"prompt reading: {tempo_prompt_s(esempio) * 1000:.1f} ms for {esempio['prompt_eval_count']} tokens")

# Output:
# generation: 343 tokens/s
# prompt reading: 13.1 ms for 11 tokens

Two things must be read carefully. The first is the separation between prompt reading and generation: a RAG context with many chunks lengthens the first phase, which on a processor without a GPU can weigh a lot, so measure both and not just the tokens generated. The second is load_duration: the first request after the model has been unloaded from memory pays for the loading, and if you leave it out of the averages you underestimate the latency the user sees. To obtain real data, call the endpoint with stream set to false and reuse the same functions, as in the following schema, which I did not run because it requires a listening server.

# Schema to adapt, NOT run: requires a listening Ollama server and a downloaded model.
import requests
from misura import token_al_secondo, tempo_prompt_s

PROMPT = "Explain in 200 words the difference between RAG and fine tuning."
r = requests.post(
    "http://127.0.0.1:11434/api/generate",
    json={"model": "mistral", "prompt": PROMPT, "stream": False},
    timeout=300,
).json()
print(token_al_secondo(r), tempo_prompt_s(r), r["load_duration"] / 1e9)

Run the schema with prompts and answer lengths that are realistic for your case, at least twenty times, and express the result as an interval, not as a single number: tokens per second vary from one request to the next. That measurement is the token_al_secondo parameter of the capacity model that follows.

A capacity model you can argue with

The simplest model that gives useful indications is a queue with a single server (M/M/1): requests arrive at random with an average frequency, the server processes them one at a time and produces a certain total number of tokens per second. If the answers are 300 tokens long on average and the server produces 20 tokens per second, each answer occupies the server for 15 seconds on average; if more requests arrive than it closes, the queue grows without limit. The formula for the mean time from request to complete answer is 1 divided by the difference between the service rate and the arrival rate, and it diverges when the two get close.

In the program the assumptions are declared at the top and are yours to replace: 300-token answers, one user asking every 120 seconds on average, and three server speeds (10, 20 and 40 tokens per second) which are test values, not measurements. The latenza_media function returns the utilisation and the mean time, or nothing if the queue is unstable; utenti_massimi looks for the maximum number of users that keeps the mean response under a target. The last calculation says how many tokens per second you need for fifty users.

"""How many users an inference server sustains, as a function of the tokens per second it produces."""


def latenza_media(utenti, pausa_s, token_risposta, token_al_secondo):
    """M/M/1 model: a single queue, one server producing 'token_al_secondo' in total.
    Returns (utilisation, mean seconds from request to complete answer) or None if unstable."""
    lam = utenti / pausa_s                    # requests per second
    mu = token_al_secondo / token_risposta    # answers completed per second at full load
    rho = lam / mu
    if rho >= 1:
        return rho, None
    return rho, 1 / (mu - lam)


def utenti_massimi(pausa_s, token_risposta, token_al_secondo, obiettivo_s):
    u = 0
    while True:
        rho, t = latenza_media(u + 1, pausa_s, token_risposta, token_al_secondo)
        if t is None or t > obiettivo_s:
            return u
        u += 1


# Assumptions, to replace with YOUR measurements: 300-token answers, each user asks every 120 s.
RISPOSTA, PAUSA = 300, 120
print("token/s  service    users with mean response within 30 s / 60 s")
for tps in (10, 20, 40):
    print(f"{tps:7d}  {RISPOSTA / tps:5.1f} s   {utenti_massimi(PAUSA, RISPOSTA, tps, 30):4d} / {utenti_massimi(PAUSA, RISPOSTA, tps, 60):4d}")

print()
print("20 token/s: how the wait grows as it approaches the limit")
for u in (2, 4, 6, 7, 8):
    rho, t = latenza_media(u, PAUSA, RISPOSTA, 20)
    print(f"{u:2d} users  utilisation {rho:4.2f}  mean response {'unstable' if t is None else f'{t:5.1f} s'}")

print()
print("requirement for 50 users (total token/s): at utilisation 1.0 and 0.5")
fabbisogno = 50 / PAUSA * RISPOSTA
print(f"{fabbisogno:.0f} and {fabbisogno / 0.5:.0f}")

# Output:
# token/s  service    users with mean response within 30 s / 60 s
#      10   30.0 s      0 /    2
#      20   15.0 s      4 /    5
#      40    7.5 s     11 /   13
#
# 20 token/s: how the wait grows as it approaches the limit
#  2 users  utilisation 0.25  mean response  20.0 s
#  4 users  utilisation 0.50  mean response  30.0 s
#  6 users  utilisation 0.75  mean response  60.0 s
#  7 users  utilisation 0.88  mean response 120.0 s
#  8 users  utilisation 1.00  mean response unstable
#
# requirement for 50 users (total token/s): at utilisation 1.0 and 0.5
# 125 and 250

The upper table tells you that at 20 tokens per second 4 users are enough to keep the mean response under 30 seconds, and 5 to keep it under 60. The second table explains why: the wait does not grow linearly, at utilisation 0.50 the mean response is 30 seconds, at 0.75 it is 60 and at 0.88 it is 120. That is why you size at half capacity, not at full capacity. The last line closes the scenario: fifty users with those assumptions require 125 tokens per second at utilisation 1.0 and 250 at utilisation 0.5, which is the healthy regime. The model is a simplification: it assumes exponential service times and a single server that processes in sequence. They serve to put orders of magnitude on the table, and must then be compared with a load test.

Memory is the second constraint, and it is computed from the model's parameters

Ollama can process several requests together for the same model, but it is not the default setting. The official FAQ lists three variables: OLLAMA_NUM_PARALLEL (parallel requests per model, default 1), OLLAMA_MAX_QUEUE (requests queued before rejecting more, default 512) and OLLAMA_MAX_LOADED_MODELS (models loaded together, default 3 for inference on the processor). It also says that the memory required scales with the product of parallel requests and context length, because a 2K context with 4 parallel requests becomes an 8K context.

• OLLAMA_NUM_PARALLEL: concurrency goes up but memory goes up too; on the processor, moreover, parallel requests share the same computation.
• OLLAMA_MAX_QUEUE: beyond this number Ollama rejects; it is your first defence against overload, but the default value (512) is too generous for answers that last 15 seconds.
• OLLAMA_MAX_LOADED_MODELS: it matters if you use two models, for example one for generation and one for embeddings.

The three settings speak at different levels: the first increases concurrency but also memory costs, the second decides what happens when the server is saturated, the third prevents two models from pushing each other out. The number you do not find in the FAQ is how much memory each additional request costs, but it can be computed. The key-value cache has, for every token, a cost that depends on the architecture: two tensors (keys and values) for every layer, for every key-value attention head, for the head dimension, for the bytes of the number. The program reads the parameters published on the Mistral-7B-Instruct-v0.3 model card and computes both the number of parameters (from the structure of the architecture) and the cache per token.

"""How much memory N parallel requests need, computed from the model's public parameters."""

# Configuration published on the Mistral-7B-Instruct-v0.3 model card (config.json, consulted on 2026-11-13)
CFG = {
    "hidden_size": 4096,
    "intermediate_size": 14336,
    "num_hidden_layers": 32,
    "num_attention_heads": 32,
    "num_key_value_heads": 8,
    "vocab_size": 32768,
}


def parametri(c):
    h, i, L = c["hidden_size"], c["intermediate_size"], c["num_hidden_layers"]
    head_dim = h // c["num_attention_heads"]
    kv = c["num_key_value_heads"] * head_dim
    attenzione = h * h + 2 * h * kv + h * h          # q, k, v, o
    mlp = 3 * h * i                                   # gate, up, down
    norme = 2 * h
    embedding = 2 * c["vocab_size"] * h               # input and output not shared
    return L * (attenzione + mlp + norme) + embedding + h


def kv_per_token(c, byte_per_valore=2):
    head_dim = c["hidden_size"] // c["num_attention_heads"]
    return 2 * c["num_hidden_layers"] * c["num_key_value_heads"] * head_dim * byte_per_valore


GIB = 1024 ** 3
n_par = parametri(CFG)
kv = kv_per_token(CFG)
print(f"parameters: {n_par / 1e9:.2f} billion")
print(f"KV cache: {kv} bytes per token ({kv / 1024:.0f} KiB)")
print(f"16-bit weights: {n_par * 2 / GIB:.1f} GiB, 4-bit (theoretical minimum): {n_par * 0.5 / GIB:.1f} GiB")
print()
print("context per request 4096 tokens; the Ollama FAQ says memory scales with parallel x context")
print("parallel    KV cache    total (4-bit weights)   total (16-bit weights)")
for parallele in (1, 2, 4, 8):
    cache = parallele * 4096 * kv / GIB
    print(f"{parallele:8d}   {cache:6.2f} GiB   {cache + n_par * 0.5 / GIB:9.1f} GiB   {cache + n_par * 2 / GIB:15.1f} GiB")

# Output:
# parameters: 7.25 billion
# KV cache: 131072 bytes per token (128 KiB)
# 16-bit weights: 13.5 GiB, 4-bit (theoretical minimum): 3.4 GiB
#
# context per request 4096 tokens; the Ollama FAQ says memory scales with parallel x context
# parallel    KV cache    total (4-bit weights)   total (16-bit weights)
#        1     0.50 GiB         3.9 GiB              14.0 GiB
#        2     1.00 GiB         4.4 GiB              14.5 GiB
#        4     2.00 GiB         5.4 GiB              15.5 GiB
#        8     4.00 GiB         7.4 GiB              17.5 GiB

The number of parameters comes out at 7.25 billion, consistent with the model's name, and the cache is 131,072 bytes per token, that is 128 KiB: a 4096-token context costs half a gibibyte per request. The credit goes to the model's 8 key-value heads (against 32 attention heads): without that reduction the cost would be fourfold. The 4-bit weights are a theoretical minimum, real quantisation formats add something, and the operating system, the runtime and the other services are not included. With eight parallel requests at 4096 tokens the cache alone weighs 4 GiB: not a chasm, but added to a server that is also a database and a vector index it stops being negligible. If you use another model, replace the dictionary with its config.json.

More processes on the same machine are not more capacity

The remedy you read most often, launching two Ollama instances on different ports behind Nginx, adds very little if the two instances sit on the same machine without a GPU: they share the same cores and the same memory bandwidth, and each loads its own copy of the weights. It is a piece of reasoning, not a measurement, so verify it with the measurement program before and after. Instances help in two cases: you have several GPUs and assign one to each instance, or you have several machines. In both cases the balancer decides where to send the requests.

For balancing between machines, open source Nginx is enough: the least_conn method sends each request to the server with the fewest active connections, and it makes sense for requests that last tens of seconds. The health check is the part that is most easily got wrong. The ngx_http_upstream_hc_module module, the one with the health_check directive, is available only with the commercial subscription, as its documentation page says; in the open source version you have the passive check, with the max_fails and fail_timeout parameters of the server in the upstream: after a certain number of errors the server is excluded for a while. The following schema is to be adapted, I did not run it because Nginx is not installed on this work machine.

# Schema to adapt, NOT run: nginx is not installed on this work machine.
upstream inferenza {
    least_conn;                      # to the connection with fewer active requests
    server 10.0.0.11:11434 max_fails=2 fail_timeout=30s;
    server 10.0.0.12:11434 max_fails=2 fail_timeout=30s;
}

server {
    listen 8080;
    location /api/generate {
        proxy_pass http://inferenza;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_buffering off;         # streamed answers arrive in pieces
        proxy_read_timeout 300s;     # a long answer is not a failure
    }
}

The two lines to read are proxy_buffering off, which lets the streamed answer through while it is being generated instead of accumulating it, and proxy_read_timeout 300s, because with a slow model a long answer easily exceeds the default timeout without anything being broken. The addresses are private example ones: never expose Ollama's port directly on the Internet, because it has no authentication.

For machines with a GPU, or for tests with two instances, the systemd template unit (ollama@.service) lets you start one instance per port: %i is the name after the at sign, so ollama@11435 uses port 11435. It is a schema to adapt and not run; note that OLLAMA_NUM_PARALLEL and OLLAMA_MAX_QUEUE are those documented in the FAQ, and that the values are examples.

# Schema to adapt, NOT run. /etc/systemd/system/ollama@.service
[Unit]
Description=Ollama, instance on port %i
After=network.target

[Service]
User=ollama
Environment="OLLAMA_HOST=127.0.0.1:%i"
Environment="OLLAMA_NUM_PARALLEL=2"
Environment="OLLAMA_MAX_QUEUE=16"
ExecStart=/usr/bin/ollama serve
Restart=on-failure

[Install]
WantedBy=multi-user.target

The queue must have a limit, or the latency has none

When requests arrive faster than the server closes them, an unbounded queue protects nobody: it accumulates waiting. A user who waits seven minutes has already reopened the page and relaunched the question, and so worsens the load. The following program simulates an hour of traffic with an arrival every 12 seconds and a service of 15 on average, that is 125% of capacity, with a random generator with a fixed seed (the two simulations are repeatable). It compares the unbounded queue with a queue that accepts at most 3 waiting requests and rejects the others.

"""Discrete-event simulation: unbounded queue against bounded queue, with more requests than the server can handle."""
import random
from statistics import quantiles


def simula(coda_max, durata=3600.0, arrivo_medio=12.0, servizio_medio=15.0, seme=42):
    rng = random.Random(seme)
    t, libero_alle = 0.0, 0.0
    attese, rifiutate = [], 0
    finiture = []                                  # service end times of the accepted requests
    while True:
        t += rng.expovariate(1 / arrivo_medio)
        if t > durata:
            break
        in_coda = sum(1 for f in finiture if f > t) - (1 if libero_alle > t else 0)
        if coda_max is not None and in_coda >= coda_max:
            rifiutate += 1
            continue
        inizio = max(t, libero_alle)
        libero_alle = inizio + rng.expovariate(1 / servizio_medio)
        finiture.append(libero_alle)
        attese.append(libero_alle - t)             # from the request to the complete answer
    q = quantiles(attese, n=20)
    return len(attese), rifiutate, quantiles(attese, n=2)[0], q[18]


print("arrivals: one every 12 s on average; service: 15 s on average (125% of capacity), 1 simulated hour")
print("queue         served  rejected   median    95th percentile")
for nome, limite in (("unbounded", None), ("max 3 waiting", 3)):
    s, r, med, p95 = simula(limite)
    print(f"{nome:14s}{s:6d}{r:10d}{med:9.0f} s{p95:12.0f} s")

# Output:
# arrivals: one every 12 s on average; service: 15 s on average (125% of capacity), 1 simulated hour
# queue         served  rejected   median    95th percentile
# unbounded        281         0      442 s         740 s
# max 3 waiting    211        83       37 s          96 s

The unbounded queue serves more requests (281 against 211) but with a median response of 442 seconds and a 95th percentile of 740. The bounded queue rejects 83 but keeps the median at 37 seconds and the high percentile at 96. It is not a universal result: it depends on the simulated arrivals and services, and the number of rejections depends on the limit you choose. The lesson is the comparison of shape: under overload, a quick rejection is worth more than a late answer, provided the rejection says when to retry. That is why OLLAMA_MAX_QUEUE at 512 is a value to lower in your case, and your application layer should answer with the 429 code and the Retry-After header.

The second level of protection is per user, not global: a single runaway script must not consume everybody's capacity. The token bucket does this in a few lines: every request costs a token, the bucket refills at a constant rate up to a maximum capacity, and whoever empties it has to wait. The program uses a clock passed as a parameter, so the logic is testable without waiting: the example user can make a burst of 5 requests and then one every 12 seconds.

"""Per-user token bucket: each request costs one token, the bucket refills over time."""


class Secchio:
    def __init__(self, capacita, gettoni_al_secondo):
        self.capacita = capacita
        self.ritmo = gettoni_al_secondo
        self.gettoni = float(capacita)
        self.ultimo = 0.0

    def consuma(self, ora, costo=1.0):
        # First refill according to the elapsed time, never beyond the capacity
        self.gettoni = min(self.capacita, self.gettoni + (ora - self.ultimo) * self.ritmo)
        self.ultimo = ora
        if self.gettoni >= costo:
            self.gettoni -= costo
            return True, 0.0
        # Seconds to wait until there is a whole token: this is the Retry-After value
        return False, (costo - self.gettoni) / self.ritmo


# One user: a burst of up to 5 requests, then one every 12 seconds (5 per minute)
secchio = Secchio(capacita=5, gettoni_al_secondo=5 / 60)
for ora in (0, 1, 2, 3, 4, 5, 6, 20, 21):
    ok, attesa = secchio.consuma(ora)
    print(f"t={ora:3d}s  {'accepted' if ok else f'rejected, retry in {attesa:4.1f} s'}")

# Output:
# t=  0s  accepted
# t=  1s  accepted
# t=  2s  accepted
# t=  3s  accepted
# t=  4s  accepted
# t=  5s  rejected, retry in  7.0 s
# t=  6s  rejected, retry in  6.0 s
# t= 20s  accepted
# t= 21s  rejected, retry in  3.0 s

In the output the first five requests go through, the sixth at second 5 is rejected with the time to wait (7 seconds, the value to put in Retry-After), and at second 20 one token has refilled. In production the bucket lives in the state shared between the instances of your service (for example Redis) and the check must be atomic; the principle does not change. A limit on the number of requests does not replace one on cost: a question with a long prompt costs much more than a short one, so you can give the request a cost proportional to the prompt's tokens.

Cache, GPU, Kubernetes: when they make sense

A response cache works when questions really repeat identically, and an internal assistant on company procedures has many of them. Two warnings: the key must include model, system prompt and parameters, otherwise you return the wrong answer to a different question; and a cache keeps text that may contain personal data, so it has a duration and must be declared in the record of processing, as the article on privacy by design treats it. A cache hit, moreover, does not enter the capacity calculation: if a third of the questions repeat, the requirement drops by a third, but you measure that, you do not assume it.

On the GPU, the rule is the same as for measurement: there is no universal acceleration factor, there is your ratio between tokens per second with and without a GPU on the same model, and the monthly cost of each from the provider's official price list at the time of purchase; prices are not reported here because they change. On Kubernetes, an orchestrator makes sense when you have several services to grow independently and a team that can manage it; for a single inference function behind a balancer, the operating cost almost always exceeds the benefit. This too is a judgement, not a datum.

What you need if capacity runs out is to know what happens when a node stops. That is the theme of the article on disaster recovery, and the opposite (when it is better to run nothing in-house) is in When NOT to self-host.

Frequently asked questions

Does Ollama handle several requests together?

Yes, for the same model, but it is off by default: OLLAMA_NUM_PARALLEL is 1. Raising it increases the memory required, which scales with parallel requests times context length, as the official FAQ says.

How many users does a server without a GPU sustain?

There is no number valid for everyone: it depends on the tokens per second it produces, the length of the answers and how often people ask. Measure the speed with Ollama's response fields and apply the capacity model; with the assumptions in the article, 20 tokens per second sustain 4-5 users with a mean response within 30-60 seconds.

Is it worth launching several Ollama instances on the same server?

Without a GPU, usually not, because the instances share the same cores and each loads its own weights into memory. They make sense with several GPUs, one per instance, or on different machines behind a balancer.

Why limit the queue?

Because an unbounded queue, under overload, makes the wait grow without end. Rejecting at once with a retry time keeps the latency of the accepted requests under control.

What you can do now

You can measure your server's speed from the fields Ollama returns (and compare two configurations with the method in the article on RAG evaluation), and turn it into a number of users with a model that declares its assumptions. You can calculate how much memory each parallel request costs from the model's public parameters, and why parallelism on the processor does not multiply capacity. You can balance several machines with open source Nginx knowing what the passive check covers and what it does not, and you can protect the service with a bounded queue and a per-user token bucket. The next step, when the load is under control, is protecting the data: disaster recovery and backup.

The same series also includes the Ollama server on a European VPS, hybrid retrieval over company documents and the recap with the decision sheet.