Ollama vs vLLM: when you use what and why the wrong choice costs 19x in latency

Ollama vs vLLM: when you use what and why the wrong choice costs 19x in latency

Ollama serves tokens at 41 TPS. vLLM reaches 793 TPS. Same GPU - an NVIDIA A100 in the Red Hat 2025 benchmark - same Llama 3.1 8B Instruct model, same quantization. The delta is 19x. It's not a marginal optimization: it is the difference between a system that supports 5 users and one that serves 500 simultaneously with p99 latency below the 100ms.

The problem is that most engineers choose the wrong tool, not out of technical ignorance, but because Ollama is so simple to start that it seems "good enough". And it is - until you put it in production with real traffic. In this episode we analyze the architectural differences between the two engines, the benchmarks updated to 2026, and an operational decision framework to know exactly when to use one and when the other. In my portfolio I have deployed Ollama on Hetzner CCX23 VPS with Llama 3.1 8B as AI provider primary: it is the right choice for that context. But if traffic grows tomorrow, I know exactly when and how migrate to vLLM.

Section 1 - Architecture comparison: llama.cpp wrapper vs PagedAttention engine

To understand the performance gap, it's not enough to look at the numbers: you need to understand why they exist. Ollama and vLLM they solve the same problem - serving inference from a Large Language Model - with radically different philosophies.

Ollama it's basically a wrapper around llama.cpp, the C++ library of Georgi Gerganov porting transformer models to consumer hardware. Its strong point is simplicity operational: weight packaging in GGUF format, OpenAI-compatible API out of the box, CPU support (with aggressive quantization), Apple Silicon via Metal, and NVIDIA GPU via CUDA. Ollama handles requests in sequential mode or with limited parallelism (maximum 4 simultaneous requests by default). There is no batching dynamic: each request occupies the GPU for its entire duration before the next one starts.

vLLM was born as a high concurrency serving system, developed by Woosuk Kwon and team at UC Berkeley and published as a peer-reviewed paper at SOSP 2023. The central contribution is PagedAttention: an algorithm that manages the KV cache (Key-Value cache of the attention mechanism) like the operating system's virtual memory - with non-contiguous logical pages and allocation on demand. The result is that vLLM reduces memory waste from 60–80% (traditional systems) to less than 4%, allowing much larger batch sizes and therefore higher throughput. On top of this, vLLM implements continuous batching: Requests are dynamically aggregated into uniform batches, maximizing GPU usage with each forward pass cycle.

The practical difference is this: Ollama processes one request at a time (or at most a small fixed pool). vLLM processes dozens or hundreds of requests in parallel, aggregating them into optimized batches. Single user the difference is minimal. With 50 concurrent users, Ollama suffocates while vLLM scales linearly.

Section 2 - Benchmarks TPS, latency and memory: the real numbers

The benchmarks below are based on the Red Hat Developer test published in August 2025, with vLLM 0.9.1 and Ollama 0.9.2 on NVIDIA A100-PCIE-40GB with Llama 3.1 8B Instruct. This is the most rigorous data available publicly with transparent methodology (load testing tool, incremental concurrency 1–256 users).

Note the single-user TTFT (Time To First Token) data: Ollama is faster by vLLM - 45ms vs 82ms. This is counterintuitive but makes sense: vLLM has startup overhead for the batching engine and it doesn't "earn" anything when there is only one request. Ollama is light and direct. The crossover point gets around 3–5 concurrent users, after which vLLM increasingly dominates.

The most important data for production decisions is not the maximum throughput, but the behavior under real load. With 50 concurrent users, Ollama builds up a queue which brings the waiting time to 3.2 seconds before generation starts. In a chatbot, this is unacceptable. In a batch processor offline, it might be ok. This is the core of the decision framework.

Section 3 - Practical Setup: Ollama in 5 Minutes

We start from Ollama because it is the natural starting point. Installation on Linux requires a single command. The following bash script installs Ollama, downloads Llama 3.1 8B and starts the server, with comments on each step to understand what happens under the hood.

# Installazione Ollama su Linux (Ubuntu/Debian)
# Il curl pipe-a bash scarica e installa il binario Ollama via script ufficiale
curl -fsSL https://ollama.com/install.sh | sh

# Ollama viene installato come servizio systemd automaticamente
# Verifica che il daemon sia attivo e in ascolto sulla porta 11434
systemctl status ollama

# Scarica il modello Llama 3.1 8B (quantizzazione Q4_K_M di default, ~5GB)
# Usa ollama run invece di ollama pull per scaricare E testare subito
ollama run llama3.1:8b

# Verifica che l'API REST sia attiva (compatibile con specifiche OpenAI)
curl http://localhost:11434/api/tags
# Output atteso: {"models":[{"name":"llama3.1:8b",...}]}

# Test veloce dell'endpoint /v1/chat/completions (formato OpenAI)
curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3.1:8b",
    "messages": [{"role": "user", "content": "Ciao, chi sei?"}],
    "stream": false
  }'

The simplicity is real: in less than 5 minutes you have an LLM that responds via API. In my portfolio I have deployed Ollama on a Hetzner CCX23 (8 vCPU, 16GB RAM, no dedicated GPU) with Llama 3.1 8B in quantization Q4_K_M - approximately 5GB of VRAM/RAM. It works, has been stable for months, and for my site traffic (less than 50 AI calls per day) is more than enough.

To configure Ollama in a more granular way, the configuration is done via environment variables or the systemd unit file. Here is the unit file annotated with the most relevant options for production:

# /etc/systemd/system/ollama.service
# Unit systemd per Ollama in produzione con parametri ottimizzati
[Unit]
Description=Ollama LLM Server
After=network-online.target
Wants=network-online.target

[Service]
# Utente dedicato senza privilegi di root (sicurezza)
User=ollama
Group=ollama
ExecStart=/usr/local/bin/ollama serve

# Variabili d'ambiente per tuning performance
# OLLAMA_HOST: bind su tutte le interfacce (default: 127.0.0.1)
Environment="OLLAMA_HOST=0.0.0.0:11434"

# OLLAMA_NUM_PARALLEL: richieste concorrenti (default: 4)
# Aumentare solo se hai VRAM sufficiente per più copie del KV cache
Environment="OLLAMA_NUM_PARALLEL=4"

# OLLAMA_MAX_QUEUE: dimensione coda in attesa (default: 512)
# Ridurre per rifiutare prima le richieste invece di accumularle
Environment="OLLAMA_MAX_QUEUE=64"

# OLLAMA_KEEP_ALIVE: tempo di idle prima di scaricare il modello (default: 5m)
# Aumentare su server dedicato per evitare cold start
Environment="OLLAMA_KEEP_ALIVE=24h"

# OLLAMA_FLASH_ATTENTION: abilita Flash Attention 2 per modelli supportati
Environment="OLLAMA_FLASH_ATTENTION=1"

Restart=always
RestartSec=3

[Install]
WantedBy=multi-user.target

After changing the unit file, reload systemd and restart the service with systemctl daemon-reload && systemctl restart ollama. The parameter OLLAMA_NUM_PARALLEL it is the most important lever: increasing it means more requests concurrent, but each parallel request uses a separate portion of VRAM for the KV cache. With an 8B model in Q4_K_M and 8GB VRAM, you can afford at most 2–3 parallels.

Section 4 - Setup vLLM with Docker: the right way

vLLM in production is almost always deployed via Docker or on Kubernetes. Create the container from scratch involves managing complex CUDA dependencies. The recommended way is to use the official image vllm/vllm-openai which already includes CUDA, PyTorch and all optimized kernels.

# Dockerfile per deployment vLLM con modello pre-scaricato nella immagine
# Base image ufficiale vLLM con CUDA 12.1 e PyTorch
FROM vllm/vllm-openai:latest

# Impostazioni ambiente per evitare prompt interattivi durante il build
ENV DEBIAN_FRONTEND=noninteractive
ENV HF_HOME=/models
ENV TRANSFORMERS_CACHE=/models

# Directory per i pesi del modello (va montata come volume esterno in produzione)
RUN mkdir -p /models

# Porta esposta: 8000 è il default di vLLM (diverso da Ollama su 11434)
EXPOSE 8000

# Comando di avvio con parametri ottimizzati per RTX 3090 / A100
# --model: path o HuggingFace model ID
# --tensor-parallel-size: numero di GPU (1 per single-GPU)
# --max-model-len: lunghezza massima del contesto in token
# --gpu-memory-utilization: percentuale VRAM usata (0.90 = 90%, lascia buffer)
# --dtype: auto sceglie BF16 su Ampere+, FP16 su GPU precedenti
CMD ["python", "-m", "vllm.entrypoints.openai.api_server", \
     "--model", "meta-llama/Llama-3.1-8B-Instruct", \
     "--tensor-parallel-size", "1", \
     "--max-model-len", "8192", \
     "--gpu-memory-utilization", "0.90", \
     "--dtype", "auto", \
     "--host", "0.0.0.0", \
     "--port", "8000"]

To launch the container with access to the GPU and model volume:

# Avvio vLLM via Docker con GPU passthrough
# --gpus all: passa tutte le GPU disponibili al container
# --shm-size: shared memory necessaria per PyTorch multiprocessing (aumentare se OOM)
# -v: mount del volume con i pesi HuggingFace (evita di riscaricarli ad ogni restart)
docker run -d \
  --name vllm-server \
  --gpus all \
  --shm-size 8g \
  -v ~/.cache/huggingface:/models \
  -p 8000:8000 \
  vllm/vllm-openai:latest \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --tensor-parallel-size 1 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.90

# Verifica che il server sia pronto (impiega 1-3 minuti al primo avvio)
# Il server è pronto quando risponde con status 200
until curl -s http://localhost:8000/health | grep -q "{}"; do
  echo "Attesa avvio vLLM..."
  sleep 5
done
echo "vLLM pronto!"

# Test endpoint modelli disponibili
curl http://localhost:8000/v1/models

The first launch requires downloading weights from HuggingFace Hub (about 15GB for Llama 3.1 8B in BF16) and compiling vLLM custom CUDA kernels (5–10 minutes). Subsequent runs are much faster because they both get cachaed. Note: To download Llama 3.1 you must have a HuggingFace token and have accepted the Meta license on huggingface.co.

Section 5 - Unified Python client: same code, interchangeable engine

One of the key advantages of both engines is the OpenAI-compatible API. This means you can write the customer code only once and switch from Ollama to vLLM by changing only the base_url. Here's a Python client that handles both with a runtime configuration:

"""
Client LLM unificato per Ollama e vLLM.
Entrambi espongono API compatibili OpenAI: cambio motore = cambio base_url.
"""
import os
import time
from openai import OpenAI
from typing import Literal

# Tipo per il motore: "ollama" o "vllm"
TipoMotore = Literal["ollama", "vllm"]


def crea_client(motore: TipoMotore = "ollama") -> tuple[OpenAI, str]:
    """
    Crea un client OpenAI configurato per Ollama o vLLM.

    Args:
        motore: "ollama" per sviluppo locale, "vllm" per produzione

    Returns:
        Tuple (client, nome_modello) pronto all'uso
    """
    if motore == "ollama":
        # Ollama: porta 11434, nessuna autenticazione necessaria
        client = OpenAI(
            base_url=os.getenv("OLLAMA_URL", "http://localhost:11434/v1"),
            api_key="ollama",  # Ollama accetta qualsiasi stringa come API key
        )
        modello = os.getenv("OLLAMA_MODEL", "llama3.1:8b")
    else:
        # vLLM: porta 8000, supporta token Bearer per autenticazione opzionale
        client = OpenAI(
            base_url=os.getenv("VLLM_URL", "http://localhost:8000/v1"),
            api_key=os.getenv("VLLM_API_KEY", "vllm-no-auth"),
        )
        modello = os.getenv("VLLM_MODEL", "meta-llama/Llama-3.1-8B-Instruct")

    return client, modello


def genera_risposta(
    prompt: str,
    motore: TipoMotore = "ollama",
    temperatura: float = 0.7,
    max_token: int = 512,
) -> dict:
    """
    Genera una risposta LLM e misura le metriche di latency.

    Returns:
        Dict con: testo, latency_ms, tokens_per_sec
    """
    client, modello = crea_client(motore)

    inizio = time.perf_counter()

    risposta = client.chat.completions.create(
        model=modello,
        messages=[{"role": "user", "content": prompt}],
        temperature=temperatura,
        max_tokens=max_token,
    )

    fine = time.perf_counter()
    latency_ms = (fine - inizio) * 1000

    testo = risposta.choices[0].message.content
    token_generati = risposta.usage.completion_tokens if risposta.usage else 0

    # Calcolo tokens per secondo (TPS)
    tps = token_generati / (latency_ms / 1000) if latency_ms > 0 else 0

    return {
        "testo": testo,
        "latency_ms": round(latency_ms, 2),
        "tokens_generati": token_generati,
        "tps": round(tps, 1),
        "motore": motore,
        "modello": modello,
    }


# Esempio di utilizzo: stesso codice, motore configurabile via env var
if __name__ == "__main__":
    motore_attivo = os.getenv("LLM_ENGINE", "ollama")  # "ollama" o "vllm"

    risultato = genera_risposta(
        prompt="Spiega PagedAttention in 3 frasi",
        motore=motore_attivo,
    )

    print(f"Motore: {risultato['motore']}")
    print(f"Latency: {risultato['latency_ms']} ms")
    print(f"Throughput: {risultato['tps']} TPS")
    print(f"Risposta: {risultato['testo'][:200]}...")

Migrating from Ollama to vLLM comes down to setup LLM_ENGINE=vllm in your configuration of production. Zero changes to the application code. This is the concrete advantage of the API OpenAI-compatible: guaranteed vendor portability.

Section 6 - Real Load Benchmarking: How to Measure Your Stack

Vendor benchmarks are useful for reference but are not a substitute for measuring on your own hardware with your real workload. The simplest tool for testing an HTTP LLM server is wrk combined with a Python script to generate realistic requests. Here's a pipeline reproducible benchmark:

#!/bin/bash
# Script benchmark per confrontare Ollama vs vLLM
# Richiede: wrk (apt install wrk) e Python 3.10+

# Configurazione test
CONCORRENZA_LIVELLI="1 5 10 25 50"
DURATA="30s"
PROMPT_FILE="benchmark_prompt.json"

# Crea il file con la richiesta di test (prompt realistico da ~50 token input)
cat > "$PROMPT_FILE" << 'EOF'
{
  "model": "llama3.1:8b",
  "messages": [
    {
      "role": "user",
      "content": "Scrivi una spiegazione tecnica di 100 parole su come funziona l'attention mechanism nei Transformer."
    }
  ],
  "max_tokens": 150,
  "temperature": 0.7,
  "stream": false
}
EOF

echo "=== BENCHMARK OLLAMA (porta 11434) ==="
for CONC in $CONCORRENZA_LIVELLI; do
  echo "--- Concorrenza: $CONC ---"
  wrk -t"$CONC" \
      -c"$CONC" \
      -d"$DURATA" \
      -s wrk_llm.lua \
      --latency \
      http://localhost:11434/v1/chat/completions
  sleep 5
done

echo "=== BENCHMARK vLLM (porta 8000) ==="
for CONC in $CONCORRENZA_LIVELLI; do
  echo "--- Concorrenza: $CONC ---"
  wrk -t"$CONC" \
      -c"$CONC" \
      -d"$DURATA" \
      -s wrk_llm.lua \
      --latency \
      http://localhost:8000/v1/chat/completions
  sleep 5
done

# Nota: wrk_llm.lua è uno script Lua che imposta Content-Type e invia il body JSON
# Crea wrk_llm.lua con:
# wrk.method = "POST"
# wrk.headers["Content-Type"] = "application/json"
# wrk.body = io.open("benchmark_prompt.json"):read("*a")

For more sophisticated benchmarks (latency percentiles, TTFT distribution, throughput in tokens/sec) the vLLM team maintains the official benchmark vllm/benchmarks/benchmark_throughput.py in the official repo that directly measures the tokens generated, not just the HTTP requests. I suggest you to use that as a verification tool after your application benchmark with wrk.

Section 7 - Decision matrix: the definitive framework

Enough theory. Here's the decision framework in code form - literally. This Python script takes your operational metrics as input and returns a reasoned recommendation:

"""
Decision framework Ollama vs vLLM.
Inserisci i parametri del tuo caso d'uso e ottieni una raccomandazione motivata.
"""
from dataclasses import dataclass
from typing import Literal


@dataclass
class ParametriCasoUso:
    """Descrive il tuo caso d'uso LLM con parametri quantitativi."""

    # Richieste medie al secondo (peak hour)
    req_per_secondo: float

    # SLA massimo accettabile per TTFT (time to first token) in millisecondi
    sla_ttft_ms: int

    # Numero di sviluppatori che gestiscono l'infrastruttura
    n_sviluppatori: int

    # Hai a disposizione una GPU NVIDIA per vLLM?
    gpu_nvidia_disponibile: bool

    # Il modello è sotto 13B di parametri? (modelli > 13B richiedono multi-GPU)
    modello_sotto_13b: bool

    # Stai facendo prototyping o è un sistema di produzione?
    fase: Literal["prototipo", "produzione"]

    # Budget mensile per infrastruttura in euro
    budget_mensile_eur: float


def raccomanda_motore(params: ParametriCasoUso) -> dict:
    """
    Applica il decision framework e restituisce raccomandazione con motivazioni.

    Il framework usa soglie derivate dai benchmark Red Hat 2025:
    - Soglia req/sec: 5 req/s è il punto dove vLLM supera nettamente Ollama
    - Soglia TTFT: sotto 200ms richiede vLLM con >5 utenti concorrenti
    """
    motore = "ollama"
    punteggio_vllm = 0
    motivazioni = []

    # Regola 1: throughput (più importante)
    if params.req_per_secondo > 10:
        punteggio_vllm += 3
        motivazioni.append(
            f"Traffico elevato ({params.req_per_secondo} req/s): "
            "vLLM è l'unica scelta praticabile oltre 10 req/s."
        )
    elif params.req_per_secondo > 5:
        punteggio_vllm += 2
        motivazioni.append(
            f"Traffico medio ({params.req_per_secondo} req/s): "
            "vLLM consigliato per SLA stabili."
        )
    else:
        motivazioni.append(
            f"Traffico basso ({params.req_per_secondo} req/s): "
            "Ollama è sufficiente con OLLAMA_NUM_PARALLEL=4."
        )

    # Regola 2: SLA latency
    if params.sla_ttft_ms < 200 and params.req_per_secondo > 3:
        punteggio_vllm += 2
        motivazioni.append(
            f"SLA aggressivo ({params.sla_ttft_ms}ms TTFT) con >3 req/s: "
            "Ollama degrada a 3200ms TTFT a 50 utenti."
        )

    # Regola 3: complessità operativa
    if params.n_sviluppatori < 2:
        motivazioni.append(
            "Team piccolo: considera il costo operativo di vLLM "
            "(tuning, monitoring, upgrade CUDA)."
        )

    # Regola 4: prerequisiti hardware
    if not params.gpu_nvidia_disponibile:
        motore = "ollama"  # vLLM senza GPU NVIDIA è impraticabile in produzione
        motivazioni.append(
            "Senza GPU NVIDIA: vLLM non è un'opzione pratica. "
            "Ollama con CPU+quantizzazione GGUF è l'alternativa."
        )
        return {"motore": motore, "motivazioni": motivazioni, "punteggio_vllm": 0}

    # Regola 5: fase di sviluppo
    if params.fase == "prototipo":
        motivazioni.append(
            "Fase prototipo: Ollama permette iterazione rapida. "
            "Puoi migrare a vLLM al momento del go-live."
        )

    # Decisione finale
    if punteggio_vllm >= 3:
        motore = "vllm"

    return {
        "motore": motore,
        "motivazioni": motivazioni,
        "punteggio_vllm": punteggio_vllm,
    }


# Esempio: chatbot interno aziendale con 20 utenti concorrenti
caso_aziendale = ParametriCasoUso(
    req_per_secondo=8.0,
    sla_ttft_ms=300,
    n_sviluppatori=2,
    gpu_nvidia_disponibile=True,
    modello_sotto_13b=True,
    fase="produzione",
    budget_mensile_eur=200.0,
)

risultato = raccomanda_motore(caso_aziendale)
print(f"Raccomandazione: {risultato['motore'].upper()}")
for m in risultato['motivazioni']:
    print(f"  - {m}")

The logic of the decision framework is transparent: there are no arbitrary thresholds. Values derive from benchmark (5 req/s is the point where Ollama degradation becomes measurable, 10 req/s is where it becomes critical). The score allows us to consider trade-offs: a system with 7 req/s and SLA of 500ms can still use Ollama if the team is one person.

Section 8 - Cost analysis: how much it costs you to get the wrong engine

The choice of engine has a direct impact on costs. Let's analyze three real-world scenarios with a calculation strict cost per token and break-even compared to cloud providers.

The cost calculation for self-hosted token is based on: GPU cost (€/h) / throughput (token/h) = €/token. With an RTX 3090 at $0.17/h and vLLM at 793 TPS, the cost per token output is approximately $0.17 / (793 × 3600) = $0.000000060 per token, which is $0.06 per million tokens. OpenAI GPT-4o output: $15 per million tokens. The ratio is 250x in favor of self-hosted.

Even with a more capable model like Llama 3.3 70B (which requires 2–4 GPUs) self-hosted costs they remain 20–50x below the cloud. The breaking point isn't volume: it's complexity operational and risk. If you have 1 developer and can't afford downtime, the cloud it still makes sense for low volumes.

Section 9 - When Ollama is the right choice also in production

The article talked a lot about when to choose vLLM, but it's important to clarify that Ollama is not for prototypes only. There are production scenarios where Ollama is the technically correct choice, not a compromise.

Scenario 1 - Inference on edge devices and heterogeneous hardware. Ollama natively supports CPU (with quantized GGUF), Apple Silicon via Metal, and AMD GPU via ROCm. If your target deployment is not NVIDIA, Ollama is often the only practical option. vLLM on CPU it exists but the performances are so degraded as to make it useless for production.

Scenario 2 - Single-tenant applications with loose SLAs. An internal tool used by a single person or a team of 5, with sequential requests and no latency requirement under 500ms, no need for vLLM. The operational overhead of vLLM (setup, tuning, monitoring) is not justified. Ollama in this context is over-engineered in the right direction: simple, stable, zero maintenance.

Scenario 3 - RAG pipeline with embedding + local generation. If your pipeline does embedding (with a template like nomic-embed-text) and generation in the same process, Ollama handles both without special configurations. Integration with pgvector for RAG on PostgreSQL is documented and stable. vLLM only handles generation, not embedding: would require a second separate service.

Scenario 4 - Rapid iteration on different models. Ollama allows you to go from Llama 3.1 to Mistral to Phi-3 with a single command (ollama run phi3). vLLM requires you to restart the container with a new configuration for each model (or to use multi-model serving with significant VRAM overhead). For teams who actively experiment with different architectures, Ollama is operationally superior.

FAQ

Are Ollama and vLLM compatible with the same API?

Yes. Both expose endpoints compatible with the OpenAI specification: /v1/chat/completions, /v1/completions e /v1/models. You can switch from Ollama to vLLM by switching only the base_url in the client without changing a line of application code. This is one of the most important practical advantages of both tools.

How many requests per second does Ollama handle before degrading?

Ollama handles up to 4 requests in parallel by default (configurable with OLLAMA_NUM_PARALLEL). With concurrency greater than 5 req/sec the requests are queued increase TTFT non-linearly: in Red Hat 2025 benchmarks, TTFT at 50 concurrent users goes up from ~45ms to ~3200ms. This is not a bug but an architectural limitation of the sequential model by llama.cpp.

Does PagedAttention also work on consumer GPUs like the RTX 3090?

Yes. vLLM with PagedAttention is compatible with any NVIDIA GPU running CUDA 11.8+, including RTX 3090 (24GB VRAM). The RTX 3090 is sufficient to serve Llama 3.1 8B in BF16 with throughput significantly higher than Ollama. Multi-GPU configurations are required for 70B+ models with tensor parallelism.

What is the real cost of an RTX 3090 GPU on the cloud?

On providers like Vast.ai and RunPod the on-demand price is $0.12–0.17/hour for RTX 3090, approximately $87–122/month continuous use. Compared to OpenAI GPT-4o output ($15/1M token), the break-even already occurs with 1–2 million tokens per month. At larger volumes the savings become substantial: 5M tokens/month = $75 OpenAI vs ~$6 self-hosted with vLLM on RTX 3090.

Does vLLM support multimodal models and function calling?

Yes. vLLM from version 0.4.x supports multimodal (vision) models such as LLaVA and InternVL, and function calling/tool use compatible with the OpenAI specification. Ollama added support experimental for tool calling starting from version 0.3.x, but with limitations on models supported and full compliance with the specification.

Can I use vLLM on CPU without GPU?

vLLM supports CPU inference via OpenVINO or llama.cpp backend but the performance is drastically lower: 5–10 tokens/sec versus 700+ on the GPU. For CPU-only deployments, Ollama with quantized GGUF (Q4_K_M or Q5_K_M) is the most practical choice, with throughput of 10–30 tokens/sec on modern server CPU.

How do you migrate from Ollama to vLLM without downtime?

Both expose the same OpenAI-compatible API, so the migration is: (1) deploy vLLM in parallel on different port (e.g. 8000 vs 11434), (2) validate the answers with an automated test suite, (3) update the base_url in your code or load balancer (nginx upstream), (4) monitor for 30 minutes, then turn off Ollama. Zero changes to the application code, zero downtime if you use a load balancer.

Conclusion

The title of the article talks about "19x latency": the real benchmark is even clearer. At 50 concurrent users, vLLM has a TTFT of 145ms versus Ollama's 3200ms. It's not a question of which is the "best" engine overall: it's a question of matching the tool to the context.

The decision framework boils down to three questions. First: How many concurrent requests are needed? Below 5 req/sec, Ollama is enough. Above 10, vLLM is required. Between 5 and 10, depends on your SLA. Second: do you have NVIDIA GPU available? Without CUDA, Ollama is the only practical choice for production. Third: how much does the system cost operationally? If you have a dedicated DevOps engineer, the overhead of vLLM is negligible. If you're a solopreneur, Ollama saves you weeks of tuning.

In my portfolio I use Ollama on Hetzner CCX23 without GPU - daily AI traffic does not justify the cost of a dedicated GPU. But I already have the customer code written with base_url configurable via env var. As traffic grows, migration will be a change of variable environment, not a refactoring.

Next episode: Ollama setup production: quantization, memory, concurrency and cost per inference. Let's delve deeper into GGUF, the sweet spot between accuracy and speed, and how Ollama integrates with it PostgreSQL + pgvector for RAG.