Self-hosted LLM stack: alternatives to Ollama 2026

1. 2026 Landscape: where inference engines stand

In early 2026, self-hosted inference splits into 4 macro-categories:

• Ollama - "Zero-config LLM" aimed at consumers and developers. Simple REST interface, built-in quantization, 1.5M+ GitHub stars. Dominant on single-machine setups.
• vLLM - "Production-grade serving engine" from Berkeley/LMSYS. PagedAttention, a breakthrough, multi-GPU parallelization, 30k+ GitHub stars. Primary scalability tier.
• llama.cpp - "Lightweight, C-only inference." Aggressive quantization (2-bit), 50kB binary, CPU-only viable. Ggml-based ecosystem.
• Text Generation Inference (TGI) - Hugging Face's production engine. Continuous batching, token streaming, Docker-native. Enterprise focus.

In 2026 the selection criterion is simple: if you are an EU SME with fewer than 100 req/sec, evaluate Ollama; if you need to scale beyond 500 req/sec or use multiple GPUs, go with vLLM; if your VRAM is constrained to under 8GB, llama.cpp is the lifesaver.

2. Ollama: the safe choice for developers (current setup)

Ollama remains the baseline for 80% of self-hosted deployments in 2026. Why? Three reasons:

• Zero-friction onboarding: `ollama pull llama3.1:8b && ollama serve` gives you a REST API on port 11434 in 2 minutes. No learning curve. Ideal for MVPs and prototypes.
• Native quantization: the GGML library is built in and selects Q4_K_M vs Q5_K_M vs FP16 based on the available VRAM. No manual configuration.
• Streaming JSON response support: the `stream: true` parameter enables token-by-token generation, which suits frontend UI/UX.

Ollama benchmark on CCX23 (4 vCPU, 16GB RAM):

Ollama verdict: best for fewer than 10 concurrent req/sec, single machine, and strict EU/Italy governance. Moderate memory footprint of 2-6GB. Minimal maintenance burden. Upgrade path to vLLM if scale is required.

3. vLLM: the power-user choice for throughput

vLLM (Berkeley LLM Serving Engine) has been around since 2023 and introduced PagedAttention, a revolution in KV cache management. Idea: treat the output tokens as RAM pages, swapping the least-used pages in and out without copying memory. Result: 20-40% latency reduction and 10x memory efficiency versus naive KV caching.

vLLM is production-grade if an EU SME requires:

• Parallel multi-query batch inference (10-50 simultaneous requests)
• Multi-GPU scaling (TP = tensor parallelism, PP = pipeline parallelism)
• Token streaming for per-token LLM feedback in the UI
• Built-in Prometheus/monitoring metrics

vLLM setup on CCX23 (a single-GPU scenario is not applicable; it requires a GPU. We simulate with a CPU-only baseline):

# Install vLLM (requires PyTorch + CUDA toolkit if using a GPU)
pip install vllm==0.4.2

# Start vLLM server (CPU fallback)
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-2-7b-hf \
  --max-num-seqs=10 \
  --max-model-len=4096 \
  --dtype float16 \
  --port 8000

# Client example
curl http://localhost:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-2-7b-hf",
    "prompt": "Explain AI in Italian",
    "max_tokens": 100,
    "temperature": 0.3,
    "stream": true
  }' | jq '.'

vLLM benchmark vs Ollama (same hardware, Llama 3.1 8B Q4):

vLLM verdict: best when batch concurrency is above 5 req/sec, for multi-GPU planning, and for EU data governance. Docker setup via Hugging Face Inference Endpoints. Trade-off: more complex configuration, a learning curve of 1-2 weeks.

4. llama.cpp: the minimalist edge case

llama.cpp (Georgi Gerganov, C++ inference) exists for one use case: VRAM under 8GB, CPU-only devices, and extreme 2-bit quantization.

Why go to such extremes? llama.cpp supports 2-bit (I2) and 3-bit quantization, which vLLM and Ollama do not yet handle robustly. Trade-off: 5-15% accuracy loss on semantic tasks, but if the workload is "classify a document into a category", 2-bit is viable.

# Build llama.cpp from source
git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
make -j$(nproc)

# Convert HF model to GGML format (Llama 3.1 8B)
python3 convert.py ../models/llama-3.1-8b

# Quantize to 2-bit (extreme compression)
./quantize ../models/llama-3.1-8b.gguf \
  ../models/llama-3.1-8b-I2.gguf I2

# Run inference server (minimal deps)
./server -m ../models/llama-3.1-8b-I2.gguf \
  -ngl 0 \
  --port 8000

# Size comparison
ls -lh ../models/llama-3.1-8b*.gguf
# F32: 33GB, Q8: 8.5GB, Q4_K_M: 4.7GB, I2: 2.1GB (!!)

llama.cpp benchmark (I2 vs Q4):

llama.cpp verdict: use it only if VRAM under 3GB is a hard constraint, or for an edge device (RPi 5, Jetson Nano). For an EU SME server scenario, the accuracy/speed trade-off is not justified. Production-ready: no (C++ maintenance burden, smaller community).

5. Text Generation Inference (TGI) by Hugging Face

TGI (Text Generation Inference) is Hugging Face's engine for production inference. It differs from vLLM in that it offers:

• Docker-first design: a pre-built image for all GPU/CPU targets, with no pip dependency hell
• Continuous batching + token streaming: compared with vLLM, its scheduler is more aggressive on dynamic batching
• OpenAI API compliance: a drop-in replacement for OpenAI client libraries (Python, JS, Go)
• Quantization via bitsandbytes: native 4-bit, 8-bit and GPTQ integration
• Native Hugging Face Hub integration: automatic model download from HF, with permissions via token

# Pull and run TGI Docker (no GPU variant)
docker run --rm \
  -p 8000:80 \
  -e MODEL_ID=meta-llama/Llama-2-7b-hf \
  -e QUANTIZE=bitsandbytes \
  ghcr.io/huggingface/text-generation-inference:2.0

# Client (Python)
from openai import OpenAI

client = OpenAI(
    api_key="whatever",
    base_url="http://localhost:8000/v1"
)

response = client.completions.create(
    model="meta-llama/Llama-2-7b-hf",
    prompt="Explain AI",
    max_tokens=100,
    stream=True
)

for chunk in response:
    print(chunk.choices[0].text, end='')

TGI benchmark (Docker, CPU baseline):

TGI verdict: best for enterprise adoption and existing OpenAI integrations. Docker containerization reduces operational friction. Backed by the Hugging Face corporate support team. Trade-off: less community activity than vLLM and fewer tutorials.

6. EU sovereignty constraints: which stack for GDPR and data residency

All 4 engines above are EU-friendly if installed on-prem at Hetzner or OVH. No cloud call-home and no data exfiltration to the US. The discriminating factor is:

• Ollama: models are downloaded once from Hugging Face (US) during setup. After that, 100% local. Personal data: NOT transmitted.
• vLLM: same as Ollama. Models from HF, inference runs locally. Prometheus/logging is optional (self-managed).
• llama.cpp: zero cloud dependencies after the build. Fully local.
• TGI: Docker image from ghcr.io (GitHub Container Registry, Microsoft US). Pulled once. Models from HF. Local afterwards.

If GDPR Article 32 (processing on-prem only) is non-negotiable, choose Ollama or llama.cpp. vLLM/TGI on a self-managed internal container is viable, but the operational overhead increases.

7. What is the optimal choice for EU SMEs in 2026?

Final decision matrix:

8. Hybrid approach: Ollama primary + vLLM failover

For risk-averse SMEs, a hybrid setup is optimal:

# Intelligent FastAPI routing
from fastapi import FastAPI
import httpx
import os

app = FastAPI()

OLLAMA_URL = os.getenv("OLLAMA_URL", "http://localhost:11434")
VLLM_URL = os.getenv("VLLM_URL", "http://vllm-gpu:8000")

@app.post("/api/query")
async def query_hybrid(request: QueryRequest):
    """
    1. Try Ollama (fast, local CPU)
    2. On timeout/error, fallback to vLLM (batch queue)
    """

    try:
        # Attempt Ollama (sub-second overhead, streaming)
        async with httpx.AsyncClient(timeout=2.0) as client:
            resp = await client.post(
                f"{OLLAMA_URL}/api/generate",
                json={
                    "model": "llama3.1:8b",
                    "prompt": request.query,
                    "stream": True
                }
            )
            return StreamingResponse(
                resp.aiter_bytes(),
                media_type="text/event-stream"
            )
    except (httpx.TimeoutException, httpx.ConnectError):
        # Fallback: vLLM (may have lower throughput but higher accuracy queue)
        print("[FALLBACK] Ollama unavailable, using vLLM queue")
        async with httpx.AsyncClient() as client:
            resp = await client.post(
                f"{VLLM_URL}/v1/completions",
                json={
                    "model": "llama-3.1-8b",
                    "prompt": request.query,
                    "max_tokens": 100,
                    "stream": True
                }
            )
            return StreamingResponse(
                resp.aiter_bytes(),
                media_type="text/event-stream"
            )

Benefits of the hybrid approach:

• Ollama as first choice: under 100ms overhead per request, deterministic
• vLLM fallback: reduces end-to-end latency when Ollama is under load
• Zero user-facing latency increase: the client receives the first token on the same timer
• Operational insurance: one system down does not mean the API is down

9. Common FAQs (12)

Q1: Can I migrate an Ollama setup to vLLM without rewriting the app?

A: Partially. vLLM does not support Ollama's JSON schema as-is. An adapter layer in FastAPI is needed (about 50 lines). See article 1280 for a deep dive on production vLLM: vLLM in production: scaling enterprise SME.

Q2: Which quantization should I choose for Ollama if VRAM is limited?

A: Ollama selects automatically. If VRAM is under 5GB, use Q4_K_M (4.7GB). If under 3GB, Q3_K_M (3.3GB, accuracy -3%). If under 2GB, llama.cpp I2 is the only option.

Q3: Does vLLM support token streaming like Ollama?

A: Yes, via SSE (Server-Sent Events). The `stream: true` parameter in the request makes the client receive tokens one by one. The UX is identical to Ollama.

Q4: Is TGI on Hetzner GDPR-compliant?

A: Yes, if the deployment is fully on-prem. Docker is pulled once, after which there are zero external calls. Hugging Face models are cached locally.

Q5: What is the annual cost comparison on a Hetzner CCX23?

A: Ollama/vLLM/TGI: €293/year (€24.49/month for a CCX23). llama.cpp: €0 on your own machine. ROI versus the OpenAI API: break-even at 50k queries/month.

Q6: Multi-language support: which engine is better for Italian RAG?

A: Ollama, vLLM and TGI are all language-agnostic. What matters is the embedding model (sentence-transformers all-MiniLM-L6-v2 supports 50+ languages). llama.cpp: the same.

Q7: Can llama.cpp run a model quantized with bitsandbytes 4-bit?

A: No. llama.cpp is GGML-native (GGUF format). Bitsandbytes is PyTorch-only (vLLM/TGI). The two ecosystems are incompatible.

Q8: Which engine is best for token counting accuracy before inference?

A: TGI: its built-in Rust tokenizer matches the exact model. vLLM: the tiktoken library. Ollama: an approximation (±5% error). This matters for latency SLA prediction.

Q9: Is vLLM tensor parallelism across 2 Hetzner GPUs worth it?

A: Yes if you run a Llama 3.1 70B model. Distributed TP reduces the memory needed per GPU. For 8B, no: a single GPU is sufficient.

Q10: Can I run Ollama + vLLM in Docker Compose containers?

A: Yes. Ollama: `docker pull ollama/ollama`, exposes port 11434. vLLM: ghcr.io image, port 8000. Compose orchestration is straightforward.

Q11: Which EU cloud provider is an alternative to Hetzner?

A: OVH (France), Contabo (Germany), Vultr (Netherlands). All offer CCX-equivalent machines. Hetzner, however, has the best price/performance in 2026.

Q12: If Ollama goes down, does the client get an error or a graceful fallback?

A: It depends on the app's implementation. I recommend a circuit breaker (Resilience4j for Java, tenacity for Python). See section 8, hybrid approach.

10. Conclusion: roadmap for choosing an engine in 2026

In 2026 EU SMEs face a varied landscape that is nonetheless converging on open-source on-prem. Ollama remains the optimal entry point for 90% of cases: simplicity, zero config, native GDPR fit, community support. vLLM becomes necessary at a scale of 50+ req/sec with a GPU available. TGI for enterprises with OpenAI integrations. llama.cpp for specific edge/mobile cases.

Suggested roadmap:

1. Start with Ollama for 2-4 weeks and validate the workload
2. Monitor latency and throughput metrics
3. If sustained load is under 5 req/sec, stay on Ollama with Redis caching
4. If 5-50 req/sec, migrate to vLLM on a single GPU
5. If over 50 req/sec, vLLM multi-GPU with TP

No choice is a "forever lock-in": the API gap between stacks is minimal (REST/OpenAI compatible). Switching cost is under 2 weeks of engineering time.