Serve Ollama in production VPS EU 2026: complete setup Hetzner

What you are really deciding when you serve a model on a European VPS

Serving a language model on your own server in Europe means making three decisions, not one: how much memory you need (and therefore which machine), how to stop the endpoint from being reachable by anyone, and how to accept that generation on a CPU is slow. Ollama, the program chosen as the example, makes installation easy; it does not make these three choices easy, and they are the ones where a small company gets it wrong.

The short answer, before the detail: a quantized model of 7-8 billion parameters fits in 8-16 GB of RAM, but on a VPS without a GPU it produces a few tens of tokens per second in the best case and a few tokens per second in the typical case. It works for a team that queries its own documents, not for a service with dozens of simultaneous users. The article gives you the calculations to check this against your own case, and states where the numbers are assumptions to be measured.

An analogy to fix the idea: choosing the machine is like choosing the van for a house move. The volume of the load (the model weights) decides whether it fits; the length of the last box you open (the context) decides whether there is room left for anything else; the speed of the van (memory bandwidth) decides how long you take. A smaller van is not slower because it is worse, but because it carries less per trip.

When it makes sense to do it yourself

The reasons that hold up, for a small or medium company, are few and need to be stated with their costs. The others, like "it costs zero in API fees", are only valid after you have counted everything else (we will do that in the article on costs).

• Data you do not want to send to a third party: contracts, medical records, minutes of meetings. If the texts stay on your server, the model provider never sees them.
• Stable, predictable volume: steady use amortizes a fixed fee better than spiky use.
• Independence from a single provider: open weights can be moved to another machine or another European provider.
• Experiments with no marginal cost: trying ten more prompts does not generate an invoice.

None of these reasons says "it is faster" or "it is more capable": a small model of your own is almost always slower and less capable than a large model in the cloud. If answer quality matters more than confidentiality, the comparison has to be made with your own data: the article on how to choose between a small and a large model shows how, and the one on when the European cloud wins lists the cases in which it is better not to do it.

How much memory you need: weights, cache and margin

The memory taken by a running model has three parts. The weights are the model file: they depend on the number of parameters and on the quantization. The K/V cache is the memory attention uses to avoid rereading the whole text at every token: it grows linearly with the context and with the number of requests processed in parallel. The margin covers compute buffers, the operating system and everything else.

For the weights it is not worth calculating by hand: the Ollama library publishes the file size of each model. Dividing that size by the number of parameters (from the model card) gives the effective bits per weight, which come out close to 4.9: the default quantization is about 4-5 bits, not exactly 4. The K/V cache, instead, is computed from the architecture: two matrices (K and V) per layer, for each of the KV heads, for the head dimension, times two bytes of 16-bit floating point. The program below does it.

"""Memory sizing for serving a model locally.

All input data comes from public sources (see the article):
- parameters and architecture: config.json and the Hugging Face API;
- size of the quantized file: the model page in the Ollama library.
Assumptions are marked ASSUMPTION.
"""
GIB = 1024 ** 3

MODELS = {
    # name: (parameters, layers, KV heads, head dim, GB of the default tag in the library)
    "llama3.1:8b": (8_030_261_248, 32, 8, 128, 4.9),
    "mistral:7b": (7_248_023_552, 32, 8, 128, 4.4),
    "qwen2.5:7b": (7_615_616_512, 28, 4, 128, 4.7),
}
KV_BYTES = 2  # ASSUMPTION: K/V cache in f16, Ollama's default
OVERHEAD_GIB = 1.0  # ASSUMPTION: compute buffers and runtime, to be measured on your own load
OS_RESERVE_GIB = 1.5  # ASSUMPTION: operating system, nginx, everything else
RAM_GB = {"CCX13": 8, "CCX23": 16, "CCX33": 32}  # Hetzner General Purpose page


def kv_per_token(layers, kv_heads, head_dim):
    # 2 = one K matrix and one V matrix for every layer
    return 2 * layers * kv_heads * head_dim * KV_BYTES


print("bits per weight implied by the downloaded file (decimal GB, assumption)")
for name, (par, st, kv, hd, gb) in MODELS.items():
    print(f"  {name:12s} {gb * 1e9 * 8 / par:5.2f} bit/weight")

print("\nKV cache per token and per 8192 tokens of context")
for name, (par, st, kv, hd, gb) in MODELS.items():
    b = kv_per_token(st, kv, hd)
    print(f"  {name:12s} {b // 1024:4d} KiB/token  {b * 8192 / GIB:5.2f} GiB at 8192")

print("\nTotal memory (weights + KV + overhead) and verdict per RAM size")
for context, parallel in [(4096, 1), (8192, 1), (8192, 4)]:
    print(f"context {context}, parallel requests {parallel}")
    for name, (par, st, kv, hd, gb) in MODELS.items():
        weights = gb * 1e9 / GIB
        cache = kv_per_token(st, kv, hd) * context * parallel / GIB
        tot = weights + cache + OVERHEAD_GIB
        verdict = []
        for srv, ram in RAM_GB.items():
            ok = tot + OS_RESERVE_GIB <= ram * 1e9 / GIB
            verdict.append(f"{srv}:{'yes' if ok else 'NO'}")
        print(f"  {name:12s} {tot:5.2f} GiB  " + " ".join(verdict))

# Output:
# bits per weight implied by the downloaded file (decimal GB, assumption)
#   llama3.1:8b   4.88 bit/weight
#   mistral:7b    4.86 bit/weight
#   qwen2.5:7b    4.94 bit/weight
#
# KV cache per token and per 8192 tokens of context
#   llama3.1:8b   128 KiB/token   1.00 GiB at 8192
#   mistral:7b    128 KiB/token   1.00 GiB at 8192
#   qwen2.5:7b     56 KiB/token   0.44 GiB at 8192
#
# Total memory (weights + KV + overhead) and verdict per RAM size
# context 4096, parallel requests 1
#   llama3.1:8b   6.06 GiB  CCX13:NO CCX23:yes CCX33:yes
#   mistral:7b    5.60 GiB  CCX13:yes CCX23:yes CCX33:yes
#   qwen2.5:7b    5.60 GiB  CCX13:yes CCX23:yes CCX33:yes
# context 8192, parallel requests 1
#   llama3.1:8b   6.56 GiB  CCX13:NO CCX23:yes CCX33:yes
#   mistral:7b    6.10 GiB  CCX13:NO CCX23:yes CCX33:yes
#   qwen2.5:7b    5.81 GiB  CCX13:yes CCX23:yes CCX33:yes
# context 8192, parallel requests 4
#   llama3.1:8b   9.56 GiB  CCX13:NO CCX23:yes CCX33:yes
#   mistral:7b    9.10 GiB  CCX13:NO CCX23:yes CCX33:yes
#   qwen2.5:7b    7.13 GiB  CCX13:NO CCX23:yes CCX33:yes

Three things can be read in the output. The first: Llama 3.1 8B and Mistral 7B pay 128 KiB of cache per token, Qwen 2.5 7B only 56 KiB, because it has 4 KV heads instead of 8 and fewer layers; at 8192 tokens the difference is half a gibibyte, negligible in one case and decisive in the other. The second: with 4 parallel requests the memory grows by about 3 GiB for Llama, because the cache is multiplied, not the weights file. The third: the 7.5 GiB usable on an 8 GB machine are just right for Mistral at a context of 4096 and not enough for Llama with the same configuration, according to the assumptions stated at the top of the program (1 GiB of overhead, 1.5 GiB for the system). They are your assumptions to measure: watch the process memory under your own load and correct the two constants.

The program does not know two facts. If the context the application requests exceeds what the model supports, the request is truncated or rejected; and if RAM is not enough, the operating system starts using swap on disk, with a slowdown you notice immediately. In both cases the first thing to change is the size of the model or the context, not the machine.

How fast it is: the ceiling set by memory bandwidth

To produce each token a dense model essentially rereads all of its weights. On a CPU the bottleneck is therefore memory bandwidth, not the number of cores: if the bandwidth is B gigabytes per second and the weights weigh P gigabytes, the speed does not exceed B/P tokens per second. It is a ceiling, not a forecast, because the actual computation costs additional time, but it tells you straight away what is impossible.

The bandwidth of a shared virtual machine is not a list-price figure: it depends on the processor and on who else uses the host. For this reason the program leaves it as an assumption and shows three values; the real measurement comes from a request to Ollama's API, whose response fields eval_count and eval_duration (nanoseconds) are documented and give tokens per second directly.

"""Theoretical speed limit and reading Ollama's metrics.

To generate a token a dense model reads (about) all of its weights from
memory. If memory bandwidth is B and the weights weigh P bytes, the speed
cannot exceed B / P tokens per second. It is a ceiling, not a forecast.
"""

WEIGHTS_GB = {"llama3.1:8b": 4.9, "mistral:7b": 4.4, "qwen2.5:3b": 1.9}
BANDWIDTHS_GB_S = [20, 40, 80]  # ASSUMPTION: the real bandwidth of a VM must be measured


def ceiling(bandwidth_gb_s, weights_gb):
    return bandwidth_gb_s / weights_gb


print("theoretical ceiling in tokens/s (assumed bandwidth -> model)")
for name, p in WEIGHTS_GB.items():
    cells = "  ".join(f"{b} GB/s: {ceiling(b, p):5.1f}" for b in BANDWIDTHS_GB_S)
    print(f"  {name:12s} {cells}")

ANSWER = 300  # tokens in a medium-length answer (ASSUMPTION)
print(f"\nminimum time for {ANSWER} tokens, llama3.1:8b")
for b in BANDWIDTHS_GB_S:
    print(f"  {b} GB/s -> {ANSWER / ceiling(b, WEIGHTS_GB['llama3.1:8b']):5.1f} s")


def tokens_per_second(response):
    """Fields documented in Ollama's API: eval_count (tokens generated) and
    eval_duration (nanoseconds spent generating them)."""
    return response["eval_count"] / (response["eval_duration"] / 1e9)


# EXAMPLE response built by hand, not measured: it only shows the formula.
example = {"eval_count": 120, "eval_duration": 15_000_000_000}
print(f"\nillustrative example: {tokens_per_second(example):.1f} tokens/s")

# Output:
# theoretical ceiling in tokens/s (assumed bandwidth -> model)
#   llama3.1:8b  20 GB/s:   4.1  40 GB/s:   8.2  80 GB/s:  16.3
#   mistral:7b   20 GB/s:   4.5  40 GB/s:   9.1  80 GB/s:  18.2
#   qwen2.5:3b   20 GB/s:  10.5  40 GB/s:  21.1  80 GB/s:  42.1
#
# minimum time for 300 tokens, llama3.1:8b
#   20 GB/s ->  73.5 s
#   40 GB/s ->  36.8 s
#   80 GB/s ->  18.4 s
#
# illustrative example: 8.0 tokens/s

With a bandwidth of 40 GB/s, Llama 3.1 8B cannot exceed about 8 tokens per second, and an answer of 300 tokens takes at least 37 seconds. A model of 3 billion parameters (here Qwen 2.5 3B, 1.9 GB) doubles or triples the ceiling, at the price of lower capability. The final example uses a response built by hand to show the formula: the real numbers are produced by your machine. On a CPU, slowness is not a configuration defect: it is the physics of the problem, and the only ways around it are a smaller model, a GPU or a job queue.

Which server: the prices that matter, with the date

Server prices change, and the ones circulating in old articles are often out of date. Hetzner's "General Purpose" page, consulted on 2026-11-13, lists for the Nuremberg location (NBG1) the values in the table, all per month and excluding VAT. The same page shows lower "from" prices for other locations and, next to some configurations, the notice "currently unavailable": availability is checked in the console, not in the article.

The comparison with the memory program is immediate: for a model of 7-8 billion parameters the 8 GB machine is at the limit, the 16 GB one is the first comfortable one, and the 32 GB one leaves room for two models or for more parallel requests. The CCX instances have dedicated vCPUs, so speed is steadier than on a shared VM, but the monthly price is an order of magnitude higher than that of a small VPS for a website: always recheck the page before doing the sums, and use the article on the costs of self-hosting to decide whether the expense pays for itself.

Installing Ollama and configuring it to stay in memory

Installation on Linux, according to the official documentation, is one command: curl -fsSL https://ollama.com/install.sh | sh. Running a downloaded script directly in a shell is an act of trust towards whoever publishes it; if your policy forbids it, the same page describes manual installation with the compressed package and a systemd unit. In any case the service listens on 127.0.0.1:11434, and the address changes only if you set OLLAMA_HOST.

The settings that matter are given through a systemd override file. The file below shows them one by one; the default values are those of Ollama's official FAQ, consulted on 2026-11-13.

# /etc/systemd/system/ollama.service.d/override.conf
# Create it with: sudo systemctl edit ollama.service
[Service]
# Leave OLLAMA_HOST at its default (127.0.0.1:11434): no open port.
# How long a model stays in memory after the last request. The default
# is 5 minutes; -1 keeps it loaded forever.
Environment="OLLAMA_KEEP_ALIVE=30m"
# Requests processed together by each model. Default: 1.
# RAM grows by OLLAMA_NUM_PARALLEL x OLLAMA_CONTEXT_LENGTH (see memory.py).
Environment="OLLAMA_NUM_PARALLEL=2"
# Models loaded together, if they fit in RAM. Default on CPU: 3.
Environment="OLLAMA_MAX_LOADED_MODELS=1"
# Requests queued before rejection with 503. Default: 512.
Environment="OLLAMA_MAX_QUEUE=32"

Every line has a reason. OLLAMA_KEEP_ALIVE decides whether the model stays in RAM between two requests: at 5 minutes (the default) the first question after a pause pays for reloading the weights from disk, at -1 the memory stays occupied even if nobody uses it. OLLAMA_NUM_PARALLEL multiplies the cache as the program above shows: doubling it without redoing the sums is the quickest way to end up in swap. OLLAMA_MAX_QUEUE limits the queue: beyond that limit the server answers 503, which for the application is a clear message instead of an endless wait. This file is a schema to adapt, not run: the variable names come from the FAQ, the values are your choices.

Protecting the endpoint: the open port is the real risk

In the FAQ and in the local API page of Ollama that I read, no authentication mechanism appears: whoever reaches the port can ask the model anything and, with the management functions, download or delete models. It is not a statement about the whole product, it is what that documentation says, and it is enough for a practical rule: the endpoint must never be exposed to the Internet without something in front of it.

The simplest solution is not to expose it at all: if the application runs on the same server, it calls 127.0.0.1. If you need to reach it from another machine, the documentation itself suggests a reverse proxy such as nginx. The schema below adds three things: a key in a header verified by nginx, a request limit, and buffering turned off for chunked responses.

# /etc/nginx/conf.d/ollama.conf  (schema to adapt, not run)
# Only needed if the application that calls the model sits on another machine.
# If it sits on the same server, it calls http://127.0.0.1:11434 and this file is not needed.

# One line per key, outside the repository, permissions 640:  "Bearer <key>" 1;
map $http_authorization $valid_key {
    default 0;
    include /etc/nginx/ollama-keys.map;
}

limit_req_zone $binary_remote_addr zone=ollama:10m rate=30r/m;

server {
    listen 443 ssl;
    server_name ai.example.eu;                  # your domain
    ssl_certificate     /etc/letsencrypt/live/ai.example.eu/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/ai.example.eu/privkey.pem;

    location / {
        if ($valid_key = 0) { return 401; }
        limit_req zone=ollama burst=5;
        proxy_pass http://127.0.0.1:11434;
        proxy_buffering off;                     # responses arrive in chunks
        proxy_read_timeout 300s;                 # a slow generation is not a failure
    }
}

The map reads the valid keys from a separate file, so the secret ends up neither in the repository nor in the text of the configuration. limit_req protects the machine from a runaway client, not from a determined attacker. proxy_buffering off is needed because a response generated token by token must reach the client as it goes, and proxy_read_timeout at 300 seconds avoids mistaking a slow generation for a failure. The schema has not been run (nginx is not installed on the machine where I wrote this article): the directives must be checked with nginx -t on your server, and you get the TLS certificate with the tool you prefer.

The firewall completes the picture: everything closed except SSH and HTTPS, and a check from the outside that port 11434 does not respond.

# Schema to adapt, not run. Open only what is needed.
sudo ufw default deny incoming
sudo ufw allow 22/tcp
sudo ufw allow 443/tcp
sudo ufw enable
# Afterwards: from the outside, port 11434 must show up as closed.
# From another machine:  nc -vz <server-ip> 11434   -> connection refused or timeout

The last line is the most important: a port that "should" be closed is checked with a connection attempt from another network, not by rereading the configuration. If you use Docker, its documentation warns that ufw and the rules created by Docker are incompatible: a port published by a container can bypass the firewall, so even then the check from outside is the only proof.

What goes wrong, in order of frequency

• The process is killed by the system for lack of memory: the model plus the cache exceed the RAM. You see it in journalctl -u ollama, the command given by the official diagnostics guide.
• The first request after a pause is very slow: the model was unloaded from memory because OLLAMA_KEEP_ALIVE is too short.
• Requests pile up and then fail with 503: the queue has reached OLLAMA_MAX_QUEUE; the load exceeds the capacity of the machine.
• The answer stops halfway in the browser: the proxy has buffering on or a timeout that is too short.
• The endpoint answers anyone: the key is missing or the port is open; it is the most serious and the quietest failure.

The last two are configuration errors that produce no messages: no log warns you that a stranger is using your CPU. The first and the second, instead, show up immediately and are fixed with the numbers from the programs above. The order matters: first close the port, then measure the memory, then optimize the speed.

Frequently asked questions

What you can do now

You can estimate, with the architecture formulas, whether a model fits the machine you are looking at and with how many users in parallel; you know which speed ceiling memory imposes and how to measure the real speed; you can configure Ollama so that it stays in RAM without saturating it and so that it does not expose an unprotected port. The next step is to decide what to run on it: the piece on document retrieval builds the part that gives the model your information, while the comparison between model families helps you choose it. The topic of capacity under load is taken up in scaling from 1 to 50 users.

The same series also includes the recap with the decision sheet.