vLLM in production: scaling enterprise SME

1. PagedAttention: KV caching revolution

The legacy problem: when generating 100 tokens from input query, attention mechanism saves Key-Value tensors for every step. On Llama 3.1 8B with 4096 token context, KV memory ≈ 1.2GB per request (bfloat16 precision). Batch 10 requests = 12GB memory wasted in inefficient caching.

Solution PagedAttention (SOSP 2023 paper): Tokenizes KV cache as virtual memory pages (e.g. 16 tokens per page). If two requests share prompt prefix, their KV pages point to the same physical RAM address (copy-on-write). When token output diverges, pages separate. Memory overhead from 12GB drops to 3-4GB batch 10.

Real impact vLLM with PagedAttention:

• Memory efficiency: Batch size 50 → 10x improvement in memory per request
• Latency p99: Scheduling + paging overhead <5ms extra vs naive batching
• Throughput: Global tokens/sec +40-60% vs sequential streaming
• Fairness: Scheduler guarantees no-starvation for long requests

2. Continuous batching: intelligent scheduling

Legacy transformer serving: wait for batch N to complete entirely before processing batch N+1. If 1 request in batch is slow (long output), entire batch is blocked. Latency p99 grows exponentially.

vLLM continuous batching: when 1 request in batch outputs token, free it from batch, insert new request from waiting list. Tokens are generated in stream, scheduler disposes dynamically.

vLLM scheduler tuning available via command-line options, supporting fcfs (default), dynamic batching, and fair-share policies.

3. Tensor Parallelism (TP) and Pipeline Parallelism (PP)

On Hetzner Cloud GPU server (2x RTX 4090, 48GB VRAM total), single GPU insufficient for Llama 3.1 70B (80GB). vLLM supports two distributed strategies:

Tensor Parallelism (TP): Split every transformer layer HORIZONTALLY across N GPU. Query + Key + Value linear projections distributed. Communication via all-reduce synchronization after each layer.

Pipeline Parallelism (PP): Split VERTICALLY: GPU1 = first 20 layers, GPU2 = last 20 layers. Less inter-GPU communication, but intra-batch latency increases (GPU1 stalls waiting for GPU2 output).

For SMEs, recommend TP for <4 GPU, PP for >4 GPU.

4. Quantization: GPTQ vs AWQ vs bitsandbytes

Reduce model from FP32 (4B per parameter) to 4-bit (0.5B per parameter) = 8x memory reduction, moderate inference latency degradation.

• GPTQ: Offline quantization, requires calibration dataset (1 epoch training). Trade-off: setup +2-4 hours, inference speedup 30%.
• AWQ: Quantizes weights based on actual activation distributions. More accurate than GPTQ, identical setup. Recommended for QA/RAG where accuracy critical.
• bitsandbytes: On-the-fly quantization, no offline prep. Trade-off: latency overhead +10-15%.

Conclusion

vLLM is production-ready for enterprise-scale self-hosted inference in 2026. PagedAttention + continuous batching + tensor parallelism + quantization = cost-effective scaling path for EU SMEs.