vLLM in produzione: scaling enterprise PMI

1. PagedAttention: rivoluzione KV caching

Il problema legacy: quando generi 100 token da input query, attenzione mechanism salva Key-Value tensors per ogni step. Su Llama 3.1 8B con context 4096 token, memoria KV ≈ 1.2GB per request (bfloat16 precision). Batch 10 richieste = 12GB memoria sprecata in caching inefficiente.

Soluzione PagedAttention (paper SOSP 2023): tokenizza KV cache come pagine di memoria virtuale (es. 16 token per pagina). Se due richieste condividono prompt prefix, le loro pagine KV puntano allo stesso indirizzo fisico RAM (copy-on-write). Quando token output diverge, le pagine si separano. Memoria overhead da 12GB scende a 3-4GB batch 10.

# Concept vLLM PagedAttention (pseudocode)
class PagedAttentionCache:
    def __init__(self, block_size=16, num_blocks=512):
        # block_size = token per pagina
        # num_blocks = max pagine in cache
        self.blocks = []  # Physical GPU memory
        self.page_table = {}  # Logical → Physical

    def allocate_pages(self, request_id, num_pages):
        """Alloca pagine fisiche, assegna a richiesta"""
        pages = self.blocks[len(self.blocks):len(self.blocks)+num_pages]
        self.page_table[request_id] = pages  # Logical mapping
        return pages

    def share_prefix(self, req1, req2, prefix_tokens):
        """Se req1 e req2 hanno stesso prompt prefix"""
        prefix_pages = req1_pages[:prefix_tokens // self.block_size]
        # req2 riusa stesse pagine fisiche, zero copy
        self.page_table[req2][:len(prefix_pages)] = prefix_pages

# Result: batch 10 richieste su Llama 8B, same prompt prefix
# Memory: 12GB (naive) → 4GB (PagedAttention), 3x savings

Impatto reale vLLM con PagedAttention:

• Memory efficiency: Batch size 50 → 10x miglioramento memoria per request
• Latency p99: Scheduling + paging overhead <5ms extra vs naive batching
• Throughput: Token/sec globale +40-60% rispetto streaming sequenziale
• Fairness: Scheduler garantisce no-starvation per requests lunghe

Verifica vLLM con PagedAttention enabled (default):

# Start vLLM server con verbose logs
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-2-7b-hf \
  --max-num-seqs 50 \
  --enable-prefix-caching \
  --gpu-memory-utilization 0.9 \
  --dtype bfloat16 \
  --port 8000 \
  --log-level debug

# Nel log, vedi:
# [INFO] Initialized KV cache with block_size=16, num_blocks=2048
# [INFO] PagedAttention enabled, prefix_caching enabled
# [INFO] Estimated GPU memory: 14GB / 16GB max

2. Continuous batching: scheduling intelligente

Legacy transformer serving: attendi che batch N termini completamente prima di processare batch N+1. Se 1 request in batch è lenta (long output), intero batch è bloccato. Latency p99 cresce esponenzialmente.

vLLM continuous batching: quando 1 request in batch output token, liberala da batch, inserisci nuova richiesta in lista waiting. Token sono generati in stream, scheduler dispone dinamicamente.

# Continuous batching in vLLM
class ContinuousScheduler:
    def __init__(self):
        self.running_queue = []  # Requests mid-generation
        self.waiting_queue = []  # Requests awaiting slot

    def schedule(self, new_request):
        """Arriva nuova request mid-batch"""
        self.waiting_queue.append(new_request)

        # Check running queue: ci sono requests che hanno finito un token?
        for req in self.running_queue[:]:
            if req.is_finished():
                self.running_queue.remove(req)
                yield req.output_tensor  # Output token

        # Spazio disponibile in batch? Pull da waiting_queue
        batch_size = len(self.running_queue)
        if batch_size < self.max_batch_size and self.waiting_queue:
            new_req = self.waiting_queue.pop(0)
            self.running_queue.append(new_req)
            print(f"[SCHED] Promoted waiting request, batch size={batch_size+1}")

# Latency benefit
# Legacy batching: p99 = 5000ms (locked by slow request)
# Continuous: p99 = 800ms (new requests don't wait for others)

vLLM scheduler configurable:

# Start con scheduler tuning
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-2-7b-hf \
  --max-num-seqs 100 \
  --max-model-len 8192 \
  --scheduling-priority fcfs \
  --enable-prefix-caching

# Opzioni scheduling: fcfs (default), vllm (dynamic batching), drf (fair share)

Benchmark continuous batching effect:

3. Tensor Parallelism (TP) e Pipeline Parallelism (PP) per multi-GPU

Su Hetzner Cloud GPU server (2x RTX 4090, 48GB VRAM totali), singolo GPU non sufficiente per Llama 3.1 70B (80GB). vLLM supporta due strategie distribuite:

Tensor Parallelism (TP): Split ogni layer del transformer ORIZZONTALMENTE su N GPU. Query + Key + Value linear projections distribuiti. Comunicazione all-reduce synchronization dopo ogni layer. Setup: vLLM `--tensor-parallel-size 2`.

Pipeline Parallelism (PP): Split VERTICALMENTE: GPU1 = primi 20 layer, GPU2 = ultimi 20 layer. Meno comunicazione inter-GPU, ma intra-batch latency aumenta (stallo GPU1 aspetta output GPU2).

Per PMI, consiglio TP per <4 GPU, PP per >4 GPU:

# 2x RTX 4090 setup: Tensor Parallelism
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-70b-instruct \
  --tensor-parallel-size 2 \
  --pipeline-parallel-size 1 \
  --max-num-seqs 64 \
  --dtype bfloat16 \
  --gpu-memory-utilization 0.95 \
  --port 8000

# 4x H100 setup: Hybrid TP=2 + PP=2
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-70b-instruct \
  --tensor-parallel-size 2 \
  --pipeline-parallel-size 2 \
  --max-num-seqs 128 \
  --dtype bfloat16 \
  --port 8000

# Verify TP allocation
# [INFO] Using tensor parallel size 2
# [INFO] Initializing model on GPU 0-1
# [GPU0] Loading layers 0-20, [GPU1] loading layers 21-40, etc.

TP benchmark (Llama 3.1 70B on 2x RTX 4090):

4. Quantization: GPTQ vs AWQ vs bitsandbytes

Ridurre modello da FP32 (4B per parametro) a 4-bit (0.5B per parametro) = 8x memoria reduction, latency moderata degradation.

GPTQ (Gradient Quantization): Offline quantization. Richiede calibration dataset (1 epoch training). Trade-off: setup time +2-4 ore, inference speedup 30%.

AWQ (Activation-aware Weights): Quantizza pesi basato su distribuzioni activation reali. Più accurato GPTQ, ma setup identico. Consigliato per QA/RAG dove accuracy critica.

bitsandbytes: On-the-fly quantization, no offline prep. Trade-off: latency overhead +10-15%.

# GPTQ quantization offline (vedi vLLM docs)
# Step 1: Generate calibration data
python -c "
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained('meta-llama/Llama-2-7b-hf')
calib_data = open('calibration_100k.txt').readlines()[:1000]
# Save as .safetensors
"

# Step 2: Quantize (richiede GPU, ~2 ore per 70B)
python -m vllm.model_executor.layers.quantization.gptq.gptq_utils \
  --model meta-llama/Llama-2-7b-hf \
  --calib-data calibration_100k.txt \
  --output ./llama-2-7b-gptq

# Step 3: Run vLLM con quantized weights
python -m vllm.entrypoints.openai.api_server \
  --model ./llama-2-7b-gptq \
  --quantization gptq \
  --port 8000

# Memory: 7B model, GPTQ 4-bit = 3.5GB (vs 14GB FP32)

5. Deploy vLLM su Hetzner: Docker Compose recipe

# docker-compose.yml per vLLM + Redis cache + Prometheus
version: '3.8'

services:
  vllm:
    image: vllm/vllm-openai:latest
    container_name: vllm-llama-8b
    ports:
      - "8000:8000"
    environment:
      - MODEL_ID=meta-llama/Llama-2-7b-hf
      - QUANTIZATION=awq
      - TENSOR_PARALLEL_SIZE=1
      - MAX_NUM_SEQS=50
      - GPU_MEMORY_UTILIZATION=0.9
      - DTYPE=bfloat16
      - PORT=8000
    volumes:
      - ./models:/root/.cache/huggingface/hub
      - ./logs:/var/log/vllm
    restart: unless-stopped
    networks:
      - llm-network

  redis:
    image: redis:7-alpine
    container_name: redis-cache
    ports:
      - "6379:6379"
    command: redis-server --maxmemory 2gb --maxmemory-policy allkeys-lru
    volumes:
      - redis-data:/data
    restart: unless-stopped
    networks:
      - llm-network

  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prom-data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
    restart: unless-stopped
    networks:
      - llm-network

  nginx:
    image: nginx:alpine
    container_name: nginx-proxy
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf
      - ./ssl:/etc/nginx/ssl
    depends_on:
      - vllm
      - prometheus
    restart: unless-stopped
    networks:
      - llm-network

volumes:
  redis-data:
  prom-data:

networks:
  llm-network:
    driver: bridge