Ollama setup production: quantization, memory, concurrency, cost per inference

Ollama setup production: quantization, memory, concurrency, cost per inference

You have installed Ollama, you've done ollama pull llama3.1:8b and the model responds. Excellent. But in production this is not enough. That pull downloads the Q4_K_M variant by default - about 5GB of VRAM - and you may not even know what that acronym means. The F16 variant of the same model weighs 16GB and goes 3x slower in tokens per second. The Q2_K variant weighs 2.8GB but degrades the quality to such an extent that the answers become inconsistent. The wrong choice in production is not a technical detail: it's the difference between a service that handles 10 concurrent users and one that crashes at the fifth.

episode we see everything you need to know to put Ollama into production seriously: how GGUF quantization works and what to choose, how to plan memory (model weights + KV cache + activation), how to configure concurrency without drowning the GPU, how to integrate Postgres with pgvector for RAG, how to deploy via systemd or Docker, and - the part that no one tells you - how to calculate the real cost per token of your self-hosted setup. At the end you have concrete numbers, not theory.

Section 1 - The Quantization GGUF: Understand the acronyms before choosing

GGUF (GPT-Generated Unified Format) is the file format used by llama.cpp - the engine under Ollama - to represent model weights in compressed form. Before GGUF existed there was GGML, then came K-quantization formats (K stands for "k-means" applied to weight clusters). The difference between Q4_0 and Q4_K_M is not just the number of bits: it's the algorithm used to select those bits. K-quant uses a calibration dataset to minimize information loss, while variants without K are simple uniform truncation. Result: at equal file size, a Q4_K_M is always better than a Q4_0.

The following table shows a comparison for an 8 billion parameter model (Llama 3.1 8B). The perplexity values are delta relative to the original FP16 on the WikiText-2 dataset - the smaller the delta, the better. The TPS (tokens per second) values are measured on an NVIDIA L4 24GB GPU with a single batch.

Note the interesting data: Q8_0 occupies almost twice the VRAM of Q4_K_M and goes almost 2x slower, but the quality gain is statistically insignificant (+0.00 ppl). This explains why the community and Ollama himself choose Q4_K_M as default. Q2_K is very fast in TPS because the weights are so compressed that the VRAM read bandwidth is minimal - but quality collapses. Attention: TPS varies significantly with batch size and context, these values are indicative for batch=1.

To download tags with Ollama, use this syntax: Note: I've translated only the given Italian text, following the provided rules. If you'd like me to translate more of the original text, please let me know!

# Scarica il modello di default (Q4_K_M - raccomandato per produzione)
ollama pull llama3.1:8b

# Scarica esplicitamente Q4_K_M (stesso risultato del default)
ollama pull llama3.1:8b-instruct-q4_K_M

# Scarica Q5_K_S se hai VRAM >8GB e vuoi qualità leggermente superiore
ollama pull llama3.1:8b-instruct-q5_K_S

# Scarica Q8_0 solo per validazione/test di qualità
ollama pull llama3.1:8b-instruct-q8_0

# Verifica i modelli presenti e la loro dimensione su disco
ollama list

# Rimuovi una variante che non usi più (libera spazio disco)
ollama rm llama3.1:8b-instruct-q8_0

portfolio ( XPHX0XPHXFederico Calò's website.), the FastAPI ml-service runs Llama 3.1 8B in Q4_K_M on Hetzner VPS CCX23 with NVIDIA L4 24GB GPU. With 5GB occupied by the model, there are 19GB available for KV cache and future expansions. I could have chosen Q8_0, but at parity of hardware I would have halved TPS without any perceivable benefit in user response times.

Section 2 - Creating a Custom Modelfile for Production

Ollama uses a Model file To define the configuration of a model: which base to use, which system prompt to load, parameters such as temperature and context. In production, creating a dedicated Modelfile allows you to standardize the configuration and version it in git. It is the equivalent of a Dockerfile for your LLM.

# File: Modelfile-production
# Modello base: llama3.1 8B con quantizzazione Q4_K_M
FROM llama3.1:8b-instruct-q4_K_M

# Parametri di inferenza ottimizzati per produzione
# temperature 0.1: risposte più deterministiche per task tecnici
PARAMETER temperature 0.1

# num_ctx: contesto in token (4096 = buon bilanciamento VRAM/qualità)
# Aumentare a 8192 se le query sono lunghe, ma raddoppia la VRAM per KV cache
PARAMETER num_ctx 4096

# num_predict: token massimi per risposta (-1 = illimitato)
PARAMETER num_predict 1024

# top_p: nucleus sampling, valori più bassi = risposte più focalizzate
PARAMETER top_p 0.9

# repeat_penalty: penalizza ripetizioni nel testo generato
PARAMETER repeat_penalty 1.1

# System prompt per il tuo caso d'uso
SYSTEM """
Sei un assistente tecnico specializzato in sviluppo software.
Rispondi in modo conciso e preciso. Usa codice quando utile.
Comunica sempre in italiano a meno che l'utente non usi un'altra lingua.
"""

# Crea il modello con nome custom per produzione
# ollama create portfolio-llm -f Modelfile-production

After creating the Modelfile, create the model with XPHX0XPHX XPHX1XPHX ollama create portfolio-llm -f Modelfile-productionFrom that moment on you can use ollama run portfolio-llm to call the API with XPHX0XPHX XPHX1XPHX the following parameters: * XPHX2XPHX * XPHX3XPHX "model": "portfolio-llm"The advantage over passing parameters through the API every time is reproducibility: all clients get the same configuration, none can "forget" to set the correct temperature.

Sezione 3 - Memory planning: model weights, KV cache, activation overhead

The most common error when planning the deployment of an LLM is thinking that the required VRAM is just the weight of the GGUF file. It's not like that. The total memory occupied by a model in inference has three components: XPHX0XPHX, which represents the size of the model itself (i.e., the number of parameters), XPHX1XPHX, which represents the size of the input data, and the size of the output data.

1. Weights of the model XPHX0XPHX XPHX1XPHXThe file GGUF itself, loaded into VRAM. For Llama 3.1 8B Q4_K_M = ~5GB.
2. KV CacheMemory for storing the attention's Keys and Values during generation. It grows linearly with context and the number of parallel requests. Formula: (Note: I preserved the formatting, punctuation, and capitalization style as in the original text) 2 * num_layers * num_heads * head_dim * context_len * batch_size * dtype_bytes
3. Overhead activationTemporary tensors during the forward pass. Typically 10-15% of the weights.

The following Python script measures the actual memory usage with psutil (CPU RAM) and nvidia-smi (VRAM GPU) while Ollama is running. Use it to calibrate your configuration before going into production.

"""
Script di monitoraggio memoria Ollama in produzione.
Misura VRAM (nvidia-smi) e RAM (psutil) durante l'inferenza.
Richiede: pip install psutil requests
"""

import subprocess
import psutil
import time
import requests
import json
from typing import Optional


def misura_vram_nvidia() -> Optional[float]:
    """
    Legge la VRAM utilizzata tramite nvidia-smi.
    Restituisce GB utilizzati, o None se nvidia-smi non è disponibile.
    """
    try:
        result = subprocess.run(
            # Formato: vram_usata_MB,vram_totale_MB
            ["nvidia-smi", "--query-gpu=memory.used,memory.total",
             "--format=csv,noheader,nounits"],
            capture_output=True, text=True, timeout=5
        )
        if result.returncode == 0:
            # Prendi solo la prima GPU (indice 0)
            linea = result.stdout.strip().split("\n")[0]
            usata_mb, totale_mb = [int(v.strip()) for v in linea.split(",")]
            return {
                "vram_usata_gb": round(usata_mb / 1024, 2),
                "vram_totale_gb": round(totale_mb / 1024, 2),
                "vram_uso_percentuale": round(usata_mb / totale_mb * 100, 1)
            }
    except (subprocess.TimeoutExpired, FileNotFoundError):
        return None
    return None


def misura_ram_sistema() -> dict:
    """
    Misura la RAM di sistema con psutil.
    """
    mem = psutil.virtual_memory()
    return {
        "ram_totale_gb": round(mem.total / (1024 ** 3), 2),
        "ram_usata_gb": round(mem.used / (1024 ** 3), 2),
        "ram_disponibile_gb": round(mem.available / (1024 ** 3), 2),
        "ram_uso_percentuale": mem.percent
    }


def benchmark_inferenza_ollama(
    model: str = "llama3.1:8b",
    prompt: str = "Spiega in 3 frasi cosa è il machine learning.",
    num_chiamate: int = 3
) -> dict:
    """
    Esegue N chiamate all'API Ollama e misura:
    - Tempo totale per risposta
    - Token generati
    - Token al secondo
    - VRAM prima e dopo
    """
    url = "http://localhost:11434/api/generate"
    risultati = []

    for i in range(num_chiamate):
        # Misura VRAM prima della chiamata
        vram_prima = misura_vram_nvidia()

        payload = {
            "model": model,
            "prompt": prompt,
            "stream": False  # Attendi risposta completa per misurare
        }

        tempo_inizio = time.time()
        try:
            resp = requests.post(url, json=payload, timeout=120)
            resp.raise_for_status()
            dati = resp.json()
        except requests.RequestException as e:
            print(f"Errore chiamata {i+1}: {e}")
            continue

        tempo_totale = time.time() - tempo_inizio

        # Misura VRAM dopo la chiamata
        vram_dopo = misura_vram_nvidia()

        # Ollama restituisce eval_count (token generati) e eval_duration (ns)
        token_generati = dati.get("eval_count", 0)
        durata_ns = dati.get("eval_duration", 1)
        tps_ollama = round(token_generati / (durata_ns / 1e9), 2)

        risultati.append({
            "chiamata": i + 1,
            "tempo_totale_sec": round(tempo_totale, 3),
            "token_generati": token_generati,
            "tps_misurato": tps_ollama,
            "vram_prima": vram_prima,
            "vram_dopo": vram_dopo
        })

        print(f"Chiamata {i+1}: {token_generati} token in {tempo_totale:.2f}s = {tps_ollama} TPS")
        time.sleep(1)  # Pausa tra chiamate per evitare throttling

    return {
        "modello": model,
        "num_chiamate": num_chiamate,
        "ram_sistema": misura_ram_sistema(),
        "risultati": risultati,
        "tps_medio": round(
            sum(r["tps_misurato"] for r in risultati) / len(risultati), 2
        ) if risultati else 0
    }


if __name__ == "__main__":
    print("=== Benchmark Ollama - Memoria e Performance ===\n")
    stats = benchmark_inferenza_ollama(
        model="llama3.1:8b",
        num_chiamate=3
    )
    print("\n--- Riepilogo ---")
    print(f"RAM sistema: {stats['ram_sistema']['ram_usata_gb']}GB / {stats['ram_sistema']['ram_totale_gb']}GB")
    print(f"TPS medio: {stats['tps_medio']}")
    print(json.dumps(stats, indent=2, ensure_ascii=False))

The key field to observe is XPHX0XPHX the one that contains the code of the product XPHX1XPHX. eval_duration 's response: it is in nanoseconds and measures only the time of token generation, excluding the prefix (prompt processing). To measure the total time including the prefix, use the difference between your request and response timestamps. In my tests on Hetzner CCX23 with L4, a 200-token prompt prefix takes around 80ms - not negligible for latencies under 200ms.

Section 4 - Concurrency: OLLAMA_NUM_PARALLEL and OLLAMA_MAX_QUEUE

Ollama, unlike vLLM, was not originally designed to handle hundreds of concurrent requests. Its engine (llama.cpp) can however manage limited parallelism with the correct environment variables. Understanding these parameters is crucial because the default may be your bottleneck without you realizing it.

OLLAMA_NUM_PARALLEL Check how many simultaneous requests a single model can handle. The default is auto: Ollama chooses 1 or 4 based on available memory. Each additional slot requires proportional memory to the context (XPHX0XPHX).OLLAMA_CONTEXT_LENGTH * 2 bytes per slotFor the KV cache. With Llama 3.1 8B Q4_K_M, context 4096 and 8 parallel slots, the KV cache occupies approximately 2 GB of additional VRAM.

OLLAMA_MAX_QUEUE It is the maximum number of pending requests before Ollama responds with an HTTP 503. The default is 512 - very high. In production with strict SLA, lower this value to 20-50 to have immediate fail-fast instead of accumulating latency.

Di seguito lo snippet per configurare il servizio systemd di Ollama con tutti i parametri di produzione:

# File: /etc/systemd/system/ollama.service.d/override.conf
# Crea con: sudo systemctl edit ollama.service
# Poi: sudo systemctl daemon-reload && sudo systemctl restart ollama

[Service]
# Directory dove sono salvati i modelli (default: ~/.ollama/models)
Environment="OLLAMA_MODELS=/opt/ollama/models"

# Porta e host di ascolto (0.0.0.0 espone a rete locale/VPN, attenzione sicurezza)
Environment="OLLAMA_HOST=127.0.0.1:11434"

# Numero di richieste parallele per modello
# Formula: OLLAMA_NUM_PARALLEL <= (VRAM_disponibile - model_size_GB) / (ctx * 2 / 1024^3)
# Es: (19GB - 5GB) / (4096 * 2 / 1e9) = 14 / 0.008 = max 14, ma usa max 8 per sicurezza
Environment="OLLAMA_NUM_PARALLEL=8"

# Coda massima: abbassa per fail-fast (503 immediato invece di attesa lunga)
Environment="OLLAMA_MAX_QUEUE=50"

# Modelli massimi caricati simultaneamente (default: 3 per GPU)
Environment="OLLAMA_MAX_LOADED_MODELS=2"

# Tempo di keep-alive modello in memoria (0 = unload subito, -1 = mai unload)
# In produzione: 10m è ragionevole per bilanciare cold start e memoria
Environment="OLLAMA_KEEP_ALIVE=10m"

# Flash attention: riduce uso VRAM ~15% senza perdita qualità
Environment="OLLAMA_FLASH_ATTENTION=1"

# KV cache quantization: riduce VRAM KV cache ulteriormente (sperimentale in v0.4+)
# Environment="OLLAMA_KV_CACHE_TYPE=q8_0"

# Log level: info per produzione, debug per troubleshooting
Environment="OLLAMA_DEBUG=0"

# Limite thread CPU (utile se condividi il server con altri processi)
# Environment="OLLAMA_NUM_THREAD=8"

After modifying the override, reload systemd and reboot: Note: I preserved the formatting, punctuation, and capitalization style of the original Italian text. sudo systemctl daemon-reload && sudo systemctl restart ollamaCheck the logs with XPHX0XPHX XPHX1XPHX journalctl -u ollama -f to check that the variables have been loaded correctly.

The following Python benchmark measures the real behavior of Ollama under concurrent load with asyncio, to verify that your OLLAMA_NUM_PARALLEL is correctly configured: Note: I preserved the formatting and punctuation as in the original text.

"""
Benchmark concorrenza Ollama con asyncio.
Simula N utenti simultanei e misura latenza e throughput reali.
Richiede: pip install aiohttp
"""

import asyncio
import aiohttp
import time
import statistics
from dataclasses import dataclass, field
from typing import List


@dataclass
class RisultatoBenchmark:
    """Risultato di una singola richiesta al benchmark."""
    utente_id: int
    latenza_ms: float
    token_generati: int
    tps: float
    successo: bool
    errore: str = ""


async def richiesta_singola(
    sessione: aiohttp.ClientSession,
    utente_id: int,
    model: str,
    prompt: str
) -> RisultatoBenchmark:
    """Esegue una singola richiesta a Ollama in modo asincrono."""
    url = "http://localhost:11434/api/generate"
    payload = {
        "model": model,
        "prompt": prompt,
        "stream": False
    }

    tempo_inizio = time.monotonic()
    try:
        async with sessione.post(url, json=payload, timeout=aiohttp.ClientTimeout(total=120)) as resp:
            if resp.status != 200:
                errore = f"HTTP {resp.status}"
                return RisultatoBenchmark(
                    utente_id=utente_id, latenza_ms=0,
                    token_generati=0, tps=0, successo=False, errore=errore
                )
            dati = await resp.json()
    except asyncio.TimeoutError:
        return RisultatoBenchmark(
            utente_id=utente_id, latenza_ms=0,
            token_generati=0, tps=0, successo=False, errore="Timeout"
        )
    except aiohttp.ClientError as e:
        return RisultatoBenchmark(
            utente_id=utente_id, latenza_ms=0,
            token_generati=0, tps=0, successo=False, errore=str(e)
        )

    latenza_ms = (time.monotonic() - tempo_inizio) * 1000
    token_generati = dati.get("eval_count", 0)
    durata_ns = dati.get("eval_duration", 1)
    tps = round(token_generati / (durata_ns / 1e9), 2)

    return RisultatoBenchmark(
        utente_id=utente_id,
        latenza_ms=round(latenza_ms, 1),
        token_generati=token_generati,
        tps=tps,
        successo=True
    )


async def benchmark_concorrenza(
    utenti_simultanei: int = 8,
    model: str = "llama3.1:8b",
    prompt: str = "Elenca 5 vantaggi del Python per data science."
) -> None:
    """
    Lancia N richieste simultanee e stampa statistiche di latenza.
    """
    print(f"\n=== Benchmark: {utenti_simultanei} utenti simultanei ===")
    print(f"Modello: {model}")

    async with aiohttp.ClientSession() as sessione:
        tempo_inizio = time.monotonic()

        # Lancia tutte le richieste in parallelo
        task_list = [
            richiesta_singola(sessione, i, model, prompt)
            for i in range(utenti_simultanei)
        ]
        risultati: List[RisultatoBenchmark] = await asyncio.gather(*task_list)

        tempo_totale = time.monotonic() - tempo_inizio

    # Calcola statistiche solo sulle richieste riuscite
    successi = [r for r in risultati if r.successo]
    fallimenti = [r for r in risultati if not r.successo]

    if successi:
        latenze = [r.latenza_ms for r in successi]
        print(f"\nRichieste riuscite: {len(successi)}/{utenti_simultanei}")
        print(f"Richieste fallite:  {len(fallimenti)}/{utenti_simultanei}")
        print(f"Tempo totale:       {tempo_totale:.2f}s")
        print(f"Latenza p50:        {statistics.median(latenze):.0f}ms")
        print(f"Latenza p95:        {sorted(latenze)[int(len(latenze)*0.95)]:.0f}ms")
        print(f"Latenza max:        {max(latenze):.0f}ms")
        print(f"TPS medio:          {statistics.mean([r.tps for r in successi]):.1f}")

    for f in fallimenti:
        print(f"  Utente {f.utente_id} fallito: {f.errore}")


if __name__ == "__main__":
    # Testa prima con 4 utenti, poi aumenta gradualmente
    for n_utenti in [1, 4, 8, 16]:
        asyncio.run(benchmark_concorrenza(utenti_simultanei=n_utenti))
        time.sleep(3)  # Lascia raffreddare tra i test

This benchmark shows the real effect of OLLAMA_NUM_PARALLEL. If you have it set to 4 and launch 8 simultaneous requests, you will see that the first 4 finish quickly and the last 4 wait in line. With OLLAMA_NUM_PARALLEL=8, all 8 start together but each goes slightly slower due to scheduling. The compromise depends on your case: low single latency (use low NUM_PARALLEL) or high total throughput (use high NUM_PARALLEL).

Section 5 - RAG with Docker Compose: Ollama + Postgres + pgvector

The RAG (Retrieval-Augmented Generation) pattern on a self-hosted stack combines Ollama for inference, an embedding model (also via Ollama) to vectorize documents, and Postgres with the pgvector extension as a vector store. It's the same stack used in my ml-service portfolio: no Redis Stack or cloud services are needed, everything runs on a single VPS.

# File: docker-compose.yml
# Stack RAG completo: Ollama + Postgres + pgvector
# Avvio: docker compose up -d
# Verifica: docker compose ps

version: "3.9"

services:
  # ----------------------------------------------------------------
  # Ollama: server LLM con GPU passthrough
  # ----------------------------------------------------------------
  ollama:
    image: ollama/ollama:latest
    container_name: ollama-prod
    restart: unless-stopped
    ports:
      - "127.0.0.1:11434:11434"  # Esponi solo su localhost, no Internet
    volumes:
      # Persistenza modelli: evita re-download a ogni riavvio
      - ollama_models:/root/.ollama
    environment:
      # Configurazione concorrenza production
      OLLAMA_NUM_PARALLEL: "4"
      OLLAMA_MAX_QUEUE: "50"
      OLLAMA_KEEP_ALIVE: "10m"
      OLLAMA_FLASH_ATTENTION: "1"
    deploy:
      resources:
        reservations:
          devices:
            # Passthrough GPU NVIDIA - richiede NVIDIA Container Toolkit
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:11434/api/tags"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 60s  # Attendi avvio Ollama prima di controllare

  # ----------------------------------------------------------------
  # Postgres con pgvector: vector store per RAG
  # ----------------------------------------------------------------
  postgres:
    image: pgvector/pgvector:pg16  # Immagine ufficiale con pgvector preinstallato
    container_name: postgres-rag
    restart: unless-stopped
    ports:
      - "127.0.0.1:5432:5432"
    environment:
      POSTGRES_DB: rag_db
      POSTGRES_USER: rag_user
      # In produzione: usa Docker secrets o variabili da .env (non committare password)
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-cambia_questa_password}
      # Ottimizzazioni Postgres per workload vettoriale
      POSTGRES_INITDB_ARGS: "--encoding=UTF-8"
    volumes:
      - postgres_data:/var/lib/postgresql/data
      - ./init-db.sql:/docker-entrypoint-initdb.d/init.sql:ro
    command: >
      postgres
        -c shared_preload_libraries=vector
        -c max_connections=100
        -c shared_buffers=256MB
        -c work_mem=64MB
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U rag_user -d rag_db"]
      interval: 10s
      timeout: 5s
      retries: 5

  # ----------------------------------------------------------------
  # Il tuo servizio applicativo (es. FastAPI ml-service)
  # ----------------------------------------------------------------
  ml-service:
    build:
      context: ./ml-service
      dockerfile: Dockerfile
    container_name: ml-service
    restart: unless-stopped
    ports:
      - "127.0.0.1:8000:8000"
    environment:
      OLLAMA_BASE_URL: "http://ollama:11434"  # Nome servizio Docker come hostname
      DATABASE_URL: "postgresql://rag_user:${POSTGRES_PASSWORD}@postgres:5432/rag_db"
    depends_on:
      ollama:
        condition: service_healthy
      postgres:
        condition: service_healthy

volumes:
  ollama_models:
    driver: local
  postgres_data:
    driver: local

Create also the file init-db.sql to initialize the pgvector extensions on the first startup: Note: I've translated only the given Italian sentence and followed the provided rules. If you need a more comprehensive translation, please provide the complete text.

# File: init-db.sql - eseguito automaticamente al primo avvio Postgres
# Abilita pgvector per la ricerca semantica
CREATE EXTENSION IF NOT EXISTS vector;

# Crea la tabella per i chunk di documenti vettorializzati
# dim=384 per modelli all-MiniLM-L6-v2, dim=1536 per modelli OpenAI-compatibili
CREATE TABLE IF NOT EXISTS document_chunks (
    id BIGSERIAL PRIMARY KEY,
    content TEXT NOT NULL,
    embedding vector(384) NOT NULL,
    metadata JSONB DEFAULT '{}',
    created_at TIMESTAMPTZ DEFAULT NOW()
);

# Indice HNSW per ricerca approssimata veloce (cosine distance)
# m=16: numero connessioni per nodo (più alto = più accurato ma più lento da costruire)
# ef_construction=64: dimensione della lista candidati durante costruzione
CREATE INDEX IF NOT EXISTS idx_chunks_embedding_hnsw
    ON document_chunks
    USING hnsw (embedding vector_cosine_ops)
    WITH (m = 16, ef_construction = 64);

Section 6 - Docker vs systemd: when to use which

Choosing between Docker and systemd to deploy Ollama is not trivial. It depends on your server's architecture, your level of familiarity with the two systems, and operational requirements. Here is an honest comparison based on direct experience on a Hetzner VPS: Note: I preserved the formatting, punctuation, and capitalization style as in the original text.

portfolio, Ollama runs on systemd on the Hetzner VPS because it is the primary AI provider for the entire application - every millisecond counts. The ml-service FastAPI instead runs separately (also systemd) and calls Ollama on localhost. For local development I use Docker Compose because it allows me to have Ollama + Postgres + pgvector with a single docker compose up.

Section 7 - Inference cost: the calculation that nobody does

When choosing self-hosted vs cloud API, the real cost per token is what matters. But almost no one calculates it correctly because they mix fixed costs (server always on) with variable costs (generated tokens). Let's see how to do it precisely.

The cost for self-hosted tokens depends on three variables: the hourly cost of the server, the tokens per second that the model generates, and the actual usage (if the server is idle 80% of the time, the real cost per token is higher). This script calculates both variants.

#!/usr/bin/env bash
# Script: costo_per_token.sh
# Calcola il costo reale per token Ollama self-hosted vs cloud API
# Uso: ./costo_per_token.sh [costo_ora_eur] [tps] [utilizzo_percentuale]
# Es:  ./costo_per_token.sh 0.90 35 0.30

set -euo pipefail

# Parametri (con default per Hetzner CCX23 GPU L4 + Llama 3.1 8B Q4_K_M)
COSTO_ORA_EUR="${1:-0.90}"       # Hetzner CCX23 GPU: ~€0.90/ora
TPS="${2:-35}"                    # Tokens al secondo misurati
UTILIZZO="${3:-0.30}"             # 30% di utilizzo medio (realistico per portfolio)

# Costi cloud di riferimento (maggio 2026, prezzi indicativi)
COSTO_GPT4_OUTPUT="0.030"         # GPT-4o: $0.030 per 1K token output
COSTO_CLAUDE_OUTPUT="0.015"       # Claude 3.5 Sonnet: $0.015 per 1K token output
COSTO_GEMINI_OUTPUT="0.010"       # Gemini 1.5 Pro: $0.010 per 1K token output

echo "========================================================"
echo "  Calcolatore Costo per Token - Self-Hosted Ollama"
echo "========================================================"
echo ""
echo "Configurazione:"
echo "  Server: €${COSTO_ORA_EUR}/ora (Hetzner CCX23 GPU L4)"
echo "  Modello: Llama 3.1 8B Q4_K_M @ ${TPS} TPS"
echo "  Utilizzo stimato: $(echo "$UTILIZZO * 100" | bc)%"
echo ""

# Calcolo costo per token con utilizzo pieno (lower bound)
# Costo per secondo = costo_ora / 3600
# Costo per token = costo_per_secondo / TPS
COSTO_TOKEN_FULL=$(echo "scale=9; $COSTO_ORA_EUR / 3600 / $TPS" | bc)
COSTO_1K_TOKEN_FULL=$(echo "scale=6; $COSTO_TOKEN_FULL * 1000" | bc)

# Calcolo costo reale con utilizzo parziale (realistic cost)
# Il costo fisso del server si spalma sui token generati effettivamente
# ore_al_giorno_attive = 24 * utilizzo
# token_al_giorno = TPS * 3600 * ore_attive
TOKEN_ORA_EFFETTIVI=$(echo "scale=0; $TPS * 3600 * $UTILIZZO" | bc)
COSTO_TOKEN_REALE=$(echo "scale=9; $COSTO_ORA_EUR / $TOKEN_ORA_EFFETTIVI" | bc)
COSTO_1K_TOKEN_REALE=$(echo "scale=6; $COSTO_TOKEN_REALE * 1000" | bc)

echo "Costo per 1000 token output:"
echo "  Self-hosted (100% utilizzo):  €${COSTO_1K_TOKEN_FULL}"
echo "  Self-hosted (${UTILIZZO}x utilizzo): €${COSTO_1K_TOKEN_REALE}"
echo ""
echo "Confronto Cloud API (prezzi indicativi maggio 2026):"
echo "  GPT-4o output:     \$${COSTO_GPT4_OUTPUT} / 1K token"
echo "  Claude 3.5 Sonnet: \$${COSTO_CLAUDE_OUTPUT} / 1K token"
echo "  Gemini 1.5 Pro:    \$${COSTO_GEMINI_OUTPUT} / 1K token"
echo ""

# Calcola break-even: quanti token al giorno giustificano il server
# Break-even = quando costo_self_hosted_giorno == costo_cloud_giorno
# costo_server_giorno = costo_ora * 24
COSTO_SERVER_GIORNO=$(echo "scale=4; $COSTO_ORA_EUR * 24" | bc)
# token_breakeven_vs_gpt4 = costo_giorno_server / (costo_gpt4_per_token)
TOKEN_BREAKEVEN=$(echo "scale=0; $COSTO_SERVER_GIORNO / ($COSTO_GPT4_OUTPUT / 1000)" | bc)

echo "Break-even vs GPT-4o:"
echo "  Costo server/giorno: €${COSTO_SERVER_GIORNO}"
echo "  Token/giorno per pareggiare: ${TOKEN_BREAKEVEN}"
echo "  Equivale a: $(echo "scale=0; $TOKEN_BREAKEVEN / $TPS / 3600" | bc) ore di inferenza continua/giorno"
echo ""
echo "Conclusione: il self-hosted conviene se generi >${TOKEN_BREAKEVEN} token/giorno."

Running this script with my VPS parameters - €0.90/hour for Hetzner CCX23 with L4, 35 TPS for Llama 3.1 8B Q4_K_M, usage at 30% - we get a cost of around €0.007 per 1000 token output. GPT-4 costs $0.030 per 1000 token output: self-hosted is more than 4x cheaper. The break-even point is at around 720,000 tokens per day - a realistic volume for an application with 50-100 active users.

Note from Federico's portfolio: The Hetzner CCX23 with L4 GPU serves the federicocalo.dev portfolio with chat functionality, article recommendations, and code analysis. With Llama 3.1 8B Q4_K_M, p50 latencies of around 400ms are achieved for responses of 100 tokens - acceptable for non-real-time use cases. For low-latency perceived chat, enable streaming ( )"stream": true Ollama API, which allows tokens to be displayed as they are generated.

Frequently Asked Questions

What is the difference between Q4_K_M and Q4_0 in Ollama?

Q4_0 is the legacy variant: uniform 4-bit truncation. Q4_K_M uses K-quantization with a calibration dataset: divide the weights into groups (blocks) and applies non-uniform quantization that minimizes the reconstruction error. With equal file size, Q4_K_M is always more accurate than Q4_0. When you see "K" in the acronym, it's always the best variant. Always use K-quant when available.

Yes, you can use Ollama without a GPU.

Yes. Ollama uses llama.cpp which supports CPU-only inference. However, the performance is drastically inferior: with Llama 3.1 8B Q4_K_M on CPU (e.g., Intel Xeon with 16 cores) expect 2-5 TPS instead of 35-50 TPS on GPU L4. For personal use/development it's acceptable; for production with SLA a GPU is required. In case of CPU-only, consider smaller models like Llama 3.2 3B or Phi-3.5 mini.

How do I know how much VRAM Ollama is using in real-time?

Use nvidia-smi -l 1 for a refresh every second of the VRAM. Alternatively, Ollama exposes metrics via API: curl http://localhost:11434/api/ps show loaded models and used VRAM for each one. For continuous monitoring, use nvtop (sudo apt install nvtop) that shows VRAM, GPU usage and processes in a TUI interface similar to htop.

Does OLLAMA_NUM_PARALLEL=8 slow down individual requests?

Yes, but slightly. Each parallel slot competes for the same GPU, so the TPS per single request drops slightly. In practice, with 8 parallel slots on L4 24GB and Llama 3.1 8B Q4_K_M, the TPS per single request drops from ~35 to ~28. The total throughput (all slots summed) rises however from 35 to ~224 TPS. This is exactly the trade-off that needs to be made: single latency vs total throughput.

The Modelfile is lost if I update Ollama?

The Modelfile is incorporated into the custom model you have created with Note: I preserved the token XPHX0XPHX exactly as-is, and followed all other rules for translation. ollama createCustom models survive Ollama updates because they are saved in XPHX0XPHX local storage XPHX1XPHX, so you can continue to use them even after the update. ~/.ollama/models/ (o OLLAMA_MODELS You can lose the source Modelfile if you don't version it in git - for this reason I always recommend keeping it in the repository.

Docker Compose with Ollama works on ARM (Apple M1/M2, Raspberry Pi)?

Yes. Ollama has multi-arch Docker images that support linux/arm64 and linux/amd64. On Apple Silicon, Ollama uses the Metal GPU backend instead of CUDA, with performance similar to a mid-range dedicated GPU. On Raspberry Pi 5 (8GB RAM, no GPU) it only works in CPU-only mode and is limited to small models (Phi-3.5 mini, Qwen 2.5 1.5B). See also EP07 of the series for lighter alternatives.

To update an Ollama model without downtime?

Ollama does not support native hot-swapping. The safe procedure is: 1) ollama pull modello:nuova-versione (download in the background without interrupting the active model), 2) check that the pull is complete with XPHX0XPHX XPHX1XPHX ollama list, 3) update the Modelfile to point to the new version, 4) ollama create with the new Modelfile, 5) run a quick test, 6) remove the old version with ollama rmDowntime is only the time it takes for Ollama to reboot at point 4, typically less than 10 seconds.

Conclusion

Quantization is not a shortcut: it's engineering. Choosing Q4_K_M instead of F16 for Llama 3.1 8B means going from 16GB to 5GB of VRAM, tripling the TPS, and losing only 0.05 points of perplexity - an invisible difference in real outputs. This choice alone can determine whether your VPS will hold up to the load or go into swap. The memory planning - weights + KV cache + activation overhead - saves you from the surprise of an OOM at 3am. And the token cost calculation gives you a number to justify self-hosted infrastructure to your CTO or client.

Concurrency with OLLAMA_NUM_PARALLEL requires calibration: there is no universally correct value. Measure your actual load with the asyncio benchmark, observe VRAM during peak, and find the balance between p95 latency and total throughput. In production, always lower OLLAMA_MAX_QUEUE (50 is a good starting point) to have fail-fast instead of an infinite queue that accumulates latency.

Next episode: Nel EP03 - vLLM infrastructure deep-dive Let's explore PagedAttention, tensor parallelism and dynamic batching: the difference between 41 TPS of Ollama and 793 TPS of vLLM explained mechanically, with a step-by-step configuration for multi-GPU.