Production safety LLM: rate limit, circuit breaker, cost cap, SLA guarantee

Production Safety LLM: Rate Limit, Circuit Breaker, Cost Cap, SLA Guarantee

The model pushes 800 tokens per second. It's an enormous power and, without any guardrails, it's also a perfect attack vector. A user with a simple script can send loop requests and consume your monthly GPU budget in under sixty seconds. This isn't science fiction: this is exactly what happens when you put online an LLM endpoint without protections.

episode we build the four pillars of production safety for a self-hosted LLM: user rate limit and model rate limit with Bucketeer4j in Spring Boot, Circuit Breaker GPU-Aware that monitors real usage of the graphics card, Cost cap per session through a counter token on Redis, and SLA guarantees With Alert Prometheus on p50/p95/p99. To close everything, a manual kill switch with a cloud provider fallback as the last line of defense.

The backend portfolio of Federico already uses Bucket4j with 120 req/min on the general API and 6 req/min on AI endpoints. In this article, we start from that base and systematically extend it to cover all predictable abuse and degradation scenarios in a real production environment.

Section 1 - Rate Limit: per-user, per-endpoint and per-model

The rate limiting in an LLM system has three independent dimensions that you need to manage separately. The first is the rate per userHow many requests per minute can an authenticated user make? The second is the rate per endpointHow many total requests does the second rule govern before your backend saturates the queue? The third, specifically for multi-model systems, is the price per modelHow many parallel requests can each model receive to avoid the 70B model monopolizing the GPU, preventing requests for the 8B model used for interactive responses.

Bucket4j implements the token bucket algorithm: each bucket has a maximum capacity of tokens and refills at a constant rate (refill rate). Each request consumes one or more tokens. When the bucket is empty, the request is rejected with 429. The library supports distributed storage on Redis, essential for multi-instance environments, through the module bucket4j-redis.

Let's see the complete configuration for Spring Boot with user-per-user rate limiting differentiated by tier and model:

// RateLimitConfig.java - Configurazione Bucket4j per tier utente e per modello
@Configuration
@EnableCaching
public class RateLimitConfig {

    // Capacità massima del bucket e velocità di ricarica per tier
    public static final Map<String, BandwidthDefinition> USER_TIERS = Map.of(
        "FREE",       new BandwidthDefinition(10,  Duration.ofMinutes(1)),  // 10 req/min
        "PRO",        new BandwidthDefinition(60,  Duration.ofMinutes(1)),  // 60 req/min
        "ENTERPRISE", new BandwidthDefinition(300, Duration.ofMinutes(1))   // 300 req/min
    );

    // Rate per-model: previene monopolizzazione GPU da modello pesante
    public static final Map<String, BandwidthDefinition> MODEL_RATES = Map.of(
        "llama3.1:70b", new BandwidthDefinition(5,  Duration.ofMinutes(1)), // 5 req/min globale
        "llama3.1:8b",  new BandwidthDefinition(200, Duration.ofMinutes(1)), // 200 req/min globale
        "mistral:7b",   new BandwidthDefinition(150, Duration.ofMinutes(1))
    );

    @Bean
    public ProxyManager<String> bucketProxyManager(RedissonClient redissonClient) {
        // Usa Redisson per storage distribuito: funziona anche su cluster multi-istanza
        return Bucket4jRedisson.casBasedProxyManager(redissonClient);
    }

    // Crea o recupera il bucket per uno specifico utente e tier
    public Bucket resolveBucketForUser(String userId, String tier,
                                        ProxyManager<String> proxyManager) {
        BandwidthDefinition def = USER_TIERS.getOrDefault(tier,
            USER_TIERS.get("FREE")); // fallback a FREE se tier sconosciuto

        BucketConfiguration config = BucketConfiguration.builder()
            .addLimit(Bandwidth.classic(
                def.capacity(),
                Refill.greedy(def.capacity(), def.period())
            ))
            .build();

        // Chiave Redis: "bucket:user:{userId}" - TTL 2x period per cleanup automatico
        return proxyManager.builder().build("bucket:user:" + userId, () -> config);
    }

    // Crea o recupera il bucket globale per un modello specifico
    public Bucket resolveBucketForModel(String modelName,
                                         ProxyManager<String> proxyManager) {
        BandwidthDefinition def = MODEL_RATES.getOrDefault(modelName,
            new BandwidthDefinition(50, Duration.ofMinutes(1))); // default 50 req/min

        BucketConfiguration config = BucketConfiguration.builder()
            .addLimit(Bandwidth.classic(
                def.capacity(),
                Refill.greedy(def.capacity(), def.period())
            ))
            .build();

        return proxyManager.builder().build("bucket:model:" + modelName, () -> config);
    }

    public record BandwidthDefinition(long capacity, Duration period) {}
}

The Spring filter that applies both rate limits in sequence, with informative headers for the client:

// LlmRateLimitFilter.java - Filtro che applica rate limit user + model in cascata
@Component
@Order(1) // eseguito prima di qualsiasi altro filtro
public class LlmRateLimitFilter extends OncePerRequestFilter {

    private final ProxyManager<String> bucketProxyManager;
    private final RateLimitConfig rateLimitConfig;
    private final UserTierService userTierService;

    @Override
    protected void doFilterInternal(HttpServletRequest request,
                                     HttpServletResponse response,
                                     FilterChain chain) throws IOException, ServletException {

        // Applica solo alle richieste verso l'endpoint LLM
        if (!request.getRequestURI().startsWith("/api/v1/ai/")) {
            chain.doFilter(request, response);
            return;
        }

        String userId = extractUserId(request); // da JWT
        String tier   = userTierService.getTier(userId);
        String model  = extractModelParam(request); // da query param o body

        // Controlla il bucket dell'utente
        Bucket userBucket = rateLimitConfig.resolveBucketForUser(
            userId, tier, bucketProxyManager);
        ConsumptionProbe userProbe = userBucket.tryConsumeAndReturnRemaining(1);

        if (!userProbe.isConsumed()) {
            long waitSeconds = userProbe.getNanosToWaitForRefill() / 1_000_000_000;
            response.setHeader("X-RateLimit-Tier", tier);
            response.setHeader("Retry-After", String.valueOf(waitSeconds));
            response.setStatus(429);
            response.getWriter().write(
                "{\"error\":\"rate_limit_exceeded\",\"tier\":\"" + tier +
                "\",\"retry_after_seconds\":" + waitSeconds + "}"
            );
            return;
        }

        // Se l'utente passa, controlla il bucket del modello
        if (model != null) {
            Bucket modelBucket = rateLimitConfig.resolveBucketForModel(
                model, bucketProxyManager);
            if (!modelBucket.tryConsume(1)) {
                response.setHeader("X-RateLimit-Model", model);
                response.setStatus(429);
                response.getWriter().write(
                    "{\"error\":\"model_capacity_exceeded\",\"model\":\"" + model + "\"}"
                );
                return;
            }
        }

        // Aggiunge header informativi prima di procedere
        response.setHeader("X-RateLimit-Remaining",
            String.valueOf(userProbe.getRemainingTokens()));
        chain.doFilter(request, response);
    }

    private String extractUserId(HttpServletRequest request) {
        // Estrae user ID dal JWT nell'header Authorization
        String token = request.getHeader("Authorization");
        if (token != null && token.startsWith("Bearer ")) {
            return jwtService.extractUserId(token.substring(7));
        }
        return request.getRemoteAddr(); // fallback a IP per richieste non autenticate
    }
}

the case of Federico's backend portfolio: the endpoint /api/v1/ai/* Use a separate bucket of 6 req/min compared to the 120 req/min of general APIs. This separation is crucial because an LLM request occupies the GPU for 2-5 seconds, while a normal REST call resolves in milliseconds. Mixing the two in the same bucket means allowing a user to saturate the 120 calls with LLM requests and completely block the service.

Section 2 - Circuit Breaker GPU-Aware

The classic circuit breaker monitors the error rate of a service. In an LLM system, however, the most common problem is not an explicit crash but a silent degradation: the GPU is at 92% utilization, requests are accommodated, latency rises to 15 seconds, but no HTTP 5xx error is raised. The circuit breaker for LLM must be GPU-aware remains the same as it is a token to be preserved exactly as-is. The rest of the translation is: "GPU-aware" translates to "GPU-aware".The system must open not only to errors but also when hardware metrics indicate a system under stress.

The approach we use here integrates pybreaker with a listener that collects NVIDIA-smi metrics every 10 seconds. When the GPU utilization exceeds 90% for at least 30 consecutive seconds, the circuit breaker enters the OPEN state and all new requests receive an immediate 503 response instead of accumulating in the queue.

# circuit_breaker.py - Circuit breaker GPU-aware per vLLM
import subprocess
import pybreaker
import asyncio
import time
from dataclasses import dataclass
from typing import Optional
import httpx

@dataclass
class GpuMetrics:
    """Snapshot delle metriche GPU correnti."""
    utilization_pct: float   # percentuale utilizzo compute
    memory_used_mb: float    # memoria VRAM usata
    temperature_c: float     # temperatura in gradi Celsius
    timestamp: float         # unix timestamp

class GpuMonitor:
    """Raccoglie metriche GPU da nvidia-smi ogni 10 secondi."""

    def __init__(self, check_interval: int = 10):
        self.check_interval = check_interval
        self.latest_metrics: Optional[GpuMetrics] = None
        self._running = False

    async def start(self):
        self._running = True
        while self._running:
            self.latest_metrics = self._collect_metrics()
            await asyncio.sleep(self.check_interval)

    def _collect_metrics(self) -> GpuMetrics:
        """Esegue nvidia-smi e parsa l'output in formato CSV."""
        result = subprocess.run([
            "nvidia-smi",
            "--query-gpu=utilization.gpu,memory.used,temperature.gpu",
            "--format=csv,noheader,nounits"
        ], capture_output=True, text=True, timeout=5)

        parts = result.stdout.strip().split(", ")
        return GpuMetrics(
            utilization_pct=float(parts[0]),
            memory_used_mb=float(parts[1]),
            temperature_c=float(parts[2]),
            timestamp=time.time()
        )

    def is_under_pressure(self, utilization_threshold: float = 90.0) -> bool:
        """True se la GPU è sopra soglia (trigger circuit breaker)."""
        if self.latest_metrics is None:
            return False  # metriche non ancora disponibili: non bloccare
        return self.latest_metrics.utilization_pct >= utilization_threshold


class LlmCircuitBreakerListener(pybreaker.CircuitBreakerListener):
    """Listener che logga le transizioni di stato del circuit breaker."""

    def state_change(self, cb, old_state, new_state):
        # Logga la transizione con contesto
        print(f"[CIRCUIT_BREAKER] {cb.name}: {old_state} -> {new_state} "
              f"at {time.strftime('%Y-%m-%dT%H:%M:%SZ')}")
        # In produzione: emetti una metrica Prometheus counter qui

    def failure(self, cb, exc):
        print(f"[CIRCUIT_BREAKER] {cb.name} failure: {exc}")


# Istanza globale del circuit breaker vLLM
vllm_breaker = pybreaker.CircuitBreaker(
    fail_max=5,          # apre dopo 5 errori consecutivi
    reset_timeout=30,    # tenta reset dopo 30 secondi in OPEN
    listeners=[LlmCircuitBreakerListener()]
)

gpu_monitor = GpuMonitor(check_interval=10)


async def call_vllm_with_breaker(
    prompt: str,
    model: str,
    vllm_base_url: str = "http://localhost:8000"
) -> dict:
    """
    Invia richiesta a vLLM con circuit breaker + GPU check combinati.
    Apre il circuito sia su errori HTTP che su GPU overcrowded.
    """
    # Controlla lo stato GPU prima ancora di chiamare il breaker
    if gpu_monitor.is_under_pressure(utilization_threshold=90.0):
        raise pybreaker.CircuitBreakerError(
            "GPU utilization >= 90%: circuit open (GPU-triggered)"
        )

    @vllm_breaker
    async def _inner_call():
        async with httpx.AsyncClient(timeout=30.0) as client:
            response = await client.post(
                f"{vllm_base_url}/v1/chat/completions",
                json={
                    "model": model,
                    "messages": [{"role": "user", "content": prompt}],
                    "max_tokens": 1024
                }
            )
            response.raise_for_status()  # 4xx/5xx triggera il breaker
            return response.json()

    return await _inner_call()

This approach combines two modes of circuit opening: the classical mode based on HTTP errors (5 consecutive failures) and the proactive mode based on GPU utilization. The result is a system that also defends against silent degradation, which is the most common case in LLM servers under heavy load.

Section 3 - Cost Cap for Session with Redis

The cost cap is different from the rate limit: it does not limit the number of requests but rather the amount of resources consumed. For an LLM, the primary resource is the number of tokens generated. Each token has a virtual cost calculated based on the hardware price: a server with GPU A100 at €2/hour that generates 800 tokens/sec has a cost per token of approximately €0.0000007 (€2 divided by 2.88 million tokens generated in one hour).

Storing the counter on Redis with a TTL of 24 hours allows for automatic resetting of the cap every day without scheduled logic. When the user exceeds their tier limit, subsequent requests receive a 429 response with a message indicating the reset time.

# config/cost_cap.yaml - Configurazione cost cap per tier utente
cost_cap:
  # Prezzi virtuali per token (in micro-euro, moltiplicati per 1000 per evitare float)
  # Basati su GPU A100 @2.00 EUR/ora, 800 token/sec throughput vLLM
  token_cost_micro_eur: 0.7   # 0.0000007 EUR per token = 0.7 micro-EUR

  tiers:
    FREE:
      max_tokens_per_session: 5000        # max 0.0035 EUR per sessione
      session_ttl_seconds: 86400          # reset ogni 24 ore
      hard_stop: true                     # rifiuta richieste oltre soglia

    PRO:
      max_tokens_per_session: 50000       # max 0.035 EUR per sessione
      session_ttl_seconds: 86400
      hard_stop: false                    # avvisa ma non blocca (soft cap)

    ENTERPRISE:
      max_tokens_per_session: 500000      # max 0.35 EUR per sessione
      session_ttl_seconds: 86400
      hard_stop: false
      alert_at_pct: 80                    # alert quando arriva all'80% del cap

  # Chiave Redis: "cost:user:{userId}:session:{date}"
  redis_key_pattern: "cost:user:{user_id}:session:{date}"
  redis_expire_jitter_seconds: 3600      # jitter ±1h per evitare reset sincronizzati

# cost_cap_middleware.py - Middleware FastAPI per controllo cost cap
import redis.asyncio as aioredis
from datetime import date
from fastapi import Request, HTTPException
from starlette.middleware.base import BaseHTTPMiddleware
import json

class CostCapMiddleware(BaseHTTPMiddleware):
    """
    Traccia i token consumati per sessione utente e applica il cost cap.
    Aggiorna il contatore DOPO la risposta per contare token reali (non stimati).
    """

    def __init__(self, app, redis_client: aioredis.Redis, cost_config: dict):
        super().__init__(app)
        self.redis = redis_client
        self.config = cost_config

    async def dispatch(self, request: Request, call_next):
        # Applica solo agli endpoint AI
        if not request.url.path.startswith("/v1/"):
            return await call_next(request)

        user_id = request.state.user_id  # iniettato da AuthMiddleware
        tier = request.state.tier        # iniettato da AuthMiddleware
        tier_config = self.config["tiers"][tier]

        # Verifica il cap PRIMA di eseguire la richiesta
        current_tokens = await self._get_current_tokens(user_id)
        max_tokens = tier_config["max_tokens_per_session"]

        if current_tokens >= max_tokens and tier_config.get("hard_stop", True):
            reset_at = self._get_reset_time()
            raise HTTPException(
                status_code=429,
                detail={
                    "error": "session_cost_cap_exceeded",
                    "used_tokens": current_tokens,
                    "max_tokens": max_tokens,
                    "tier": tier,
                    "reset_at_utc": reset_at
                }
            )

        # Avviso soft cap per tier PRO/Enterprise
        alert_pct = tier_config.get("alert_at_pct", 100)
        if current_tokens >= (max_tokens * alert_pct / 100):
            request.state.cost_warning = True

        # Esegui la richiesta LLM
        response = await call_next(request)

        # Aggiorna contatore con token reali dalla risposta
        if hasattr(request.state, "tokens_used"):
            tokens_used = request.state.tokens_used  # iniettato da LlmResponseHandler
            await self._increment_tokens(user_id, tokens_used, tier_config)

        return response

    async def _get_current_tokens(self, user_id: str) -> int:
        key = f"cost:user:{user_id}:session:{date.today().isoformat()}"
        value = await self.redis.get(key)
        return int(value) if value else 0

    async def _increment_tokens(self, user_id: str, tokens: int, tier_config: dict):
        key = f"cost:user:{user_id}:session:{date.today().isoformat()}"
        ttl = tier_config["session_ttl_seconds"]
        # INCR atomico + EXPIRE: anche con richieste concorrenti il contatore è consistente
        pipe = self.redis.pipeline()
        pipe.incrby(key, tokens)
        pipe.expire(key, ttl)
        await pipe.execute()

    def _get_reset_time(self) -> str:
        from datetime import datetime, timedelta, timezone
        tomorrow = datetime.now(timezone.utc).replace(
            hour=0, minute=0, second=0, microsecond=0
        ) + timedelta(days=1)
        return tomorrow.isoformat()

Section 4 - SLA Guarantee: target and alert Prometheus

Defining an SLA without measuring it is a pointless exercise. First of all, you need to have clear objectives on which percentiles you want to guarantee. The following table shows realistic targets for various deployment scenarios, based on real benchmark results from vLLM on consumer hardware and data center: | Scenario | Target (%) | | --- | --- | | Consumer Hardware | 95% | | Data Center | 98% | Note: SLA stands for Service Level Agreement, vLLM stands for Very Large Language Model.

TTFT (Time To First Token) is the most critical metric for user experience in an interactive chatbot: it represents the time the user waits before seeing the first word. The p99 E2E (end-to-end for a complete response of 512 tokens) is more relevant for batch tasks or analysis. Define separate objectives for both.

Alerts from Prometheus fire when the SLA violation persists for at least 2 minutes, avoiding false positives due to brief moments during the warm-up phase or occasional batch bursts:

# prometheus/alerts/llm_sla.yaml - Alert rules per SLA p99 vLLM
groups:
  - name: llm_sla_alerts
    interval: 30s
    rules:

      # Alert: TTFT p99 supera 500ms per 2 minuti (scenario RTX 3090)
      - alert: LlmTtftP99High
        expr: |
          histogram_quantile(
            0.99,
            sum by (le, model) (
              rate(vllm_time_to_first_token_seconds_bucket[5m])
            )
          ) > 0.5
        for: 2m
        labels:
          severity: warning
          team: ai-infra
        annotations:
          summary: "vLLM TTFT p99 {{ $labels.model }} supera 500ms"
          description: |
            Il 99° percentile del Time To First Token per il modello
            {{ $labels.model }} è {{ $value | humanizeDuration }}.
            SLA target: 500ms. Verificare GPU utilization e coda richieste.

      # Alert: latency E2E p99 supera 8 secondi
      - alert: LlmE2eLatencyP99Critical
        expr: |
          histogram_quantile(
            0.99,
            sum by (le) (
              rate(vllm_e2e_request_latency_seconds_bucket[5m])
            )
          ) > 8.0
        for: 2m
        labels:
          severity: critical
          team: ai-infra
        annotations:
          summary: "vLLM E2E latency p99 supera 8 secondi - SLA violato"
          description: "p99={{ $value | humanizeDuration }}. Aprire circuit breaker manuale se persiste."

      # Alert: GPU utilization sopra 90% per più di 3 minuti
      - alert: GpuHighUtilization
        expr: |
          avg_over_time(
            nvidia_smi_utilization_gpu_ratio[3m]
          ) > 0.90
        for: 3m
        labels:
          severity: warning
          team: ai-infra
        annotations:
          summary: "GPU utilization media >90% da 3 minuti"
          description: "GPU={{ $labels.gpu }}, util={{ $value | humanizePercentage }}. Considerare circuit breaker."

      # Alert: coda richieste in attesa supera 50 elementi
      - alert: LlmQueueBackpressure
        expr: vllm_num_requests_waiting > 50
        for: 1m
        labels:
          severity: warning
          team: ai-infra
        annotations:
          summary: "Coda vLLM: {{ $value }} richieste in attesa"
          description: "Possibile saturazione GPU o configurazione batch subottimale."

Section 5 - Queue vs Reject-Fast: The Decisive Trade-off

When the system is overloaded, you must choose between two fundamentally different strategies: putting requests in line or rejecting them immediately. There is no universally correct answer: it depends on the specific use case and the type of experience you want to offer users.

La tail strategy It is just when your user is a batch process (batch processing, pipeline analysis, asynchronous webhook) that can wait minutes or hours without compromising the experience. You are essentially transforming synchronous requests to asynchronous ones: the client receives an immediate job ID and performs polling for the result.

La fast rejection strategy It's just right when you have an interactive latency SLA (chatbot, copilot, real-time completion). A user waiting 30 seconds for a response has already disconnected, so the queue is pointless: it's better to immediately inform the error and allow the client to display a backoff message.

# request_router.py - Router che sceglie queue o reject-fast per tipo richiesta
import asyncio
import redis.asyncio as aioredis
from enum import Enum
from dataclasses import dataclass
from typing import Optional
import uuid
import json

class RequestPriority(Enum):
    INTERACTIVE = "interactive"  # chatbot, copilot - usa reject-fast
    BATCH = "batch"              # analisi, summarization - usa coda

@dataclass
class QueueConfig:
    max_queue_size: int = 20        # rifiuta se coda > 20 (reject-fast oltre soglia)
    max_wait_seconds: int = 120     # TTL job in coda
    poll_interval_seconds: float = 0.5

async def route_request(
    request_payload: dict,
    priority: RequestPriority,
    redis_client: aioredis.Redis,
    queue_config: QueueConfig,
    vllm_client
) -> dict:
    """
    Smista la richiesta verso coda o reject-fast in base alla priorità.
    """
    if priority == RequestPriority.INTERACTIVE:
        # REJECT-FAST: tenta esecuzione immediata, fallisce veloce se sistema pieno
        return await _execute_reject_fast(request_payload, vllm_client)

    elif priority == RequestPriority.BATCH:
        # QUEUE: inserisce in coda Redis, restituisce job ID per polling
        return await _enqueue_request(request_payload, redis_client, queue_config)


async def _execute_reject_fast(payload: dict, vllm_client) -> dict:
    """
    Tenta l'esecuzione con timeout aggressivo (5s).
    Se il sistema non risponde, lancia immediatamente l'errore.
    """
    try:
        # timeout=5s: se vLLM non inizia a rispondere entro 5s, circuit aperto
        result = await asyncio.wait_for(
            vllm_client.generate(payload),
            timeout=5.0
        )
        return {"status": "ok", "result": result}
    except asyncio.TimeoutError:
        return {
            "status": "error",
            "error": "service_busy",
            "message": "Il modello è momentaneamente sovraccarico. Riprova tra 10 secondi.",
            "retry_after_seconds": 10
        }


async def _enqueue_request(
    payload: dict,
    redis_client: aioredis.Redis,
    config: QueueConfig
) -> dict:
    """
    Inserisce la richiesta in una coda Redis FIFO.
    Restituisce job_id per polling asincrono.
    """
    # Controlla dimensione coda prima di inserire
    queue_length = await redis_client.llen("llm:batch:queue")
    if queue_length >= config.max_queue_size:
        return {
            "status": "error",
            "error": "queue_full",
            "message": f"Coda batch piena ({queue_length}/{config.max_queue_size}). Riprova tra qualche minuto.",
            "current_queue_depth": queue_length
        }

    job_id = str(uuid.uuid4())
    job = {
        "job_id": job_id,
        "payload": payload,
        "enqueued_at": asyncio.get_event_loop().time()
    }

    # LPUSH + EXPIRE per TTL automatico
    await redis_client.lpush("llm:batch:queue", json.dumps(job))

    return {
        "status": "queued",
        "job_id": job_id,
        "estimated_wait_seconds": queue_length * 3,  # stima: 3s per job in coda
        "poll_url": f"/api/v1/ai/jobs/{job_id}/status"
    }

Section 6 - Kill Switch and Fallback Cloud

The kill switch is the final security layer: a manual (or automatic as last resort) mechanism that redirects all LLM traffic to a cloud provider when the self-hosted infrastructure is compromised. The secure port of Federico's team is configured to use Ollama on VPS as primary provider: any fallback to external providers is an explicit, non-automatic decision to maintain control over costs.

The underlying middleware implements a flag on Redis that can be activated via internal API (called by the team or from a Prometheus alert in auto-trigger mode with very conservative thresholds). When the flag is active, every request is instructed to OpenAI with the same response structure, making the fallback transparent to the client.

# kill_switch_middleware.py - Middleware kill switch con fallback OpenAI
import redis.asyncio as aioredis
import httpx
import os
from fastapi import Request, Response
from starlette.middleware.base import BaseHTTPMiddleware

KILL_SWITCH_KEY = "llm:kill_switch:active"
OPENAI_FALLBACK_URL = "https://api.openai.com/v1/chat/completions"

class KillSwitchMiddleware(BaseHTTPMiddleware):
    """
    Controlla il flag kill switch su Redis.
    Se attivo, redirige le richieste LLM a OpenAI come fallback cloud.
    NOTA: il flag deve essere attivato esplicitamente dal team via API interna.
    OpenAI non è mai il provider primario (vedi feedback_ai_provider_ollama_only).
    """

    def __init__(self, app, redis_client: aioredis.Redis):
        super().__init__(app)
        self.redis = redis_client
        self._cache_ttl = 5     # ri-controlla Redis ogni 5 secondi (evita overhead)
        self._cached_state = False
        self._cache_ts = 0.0

    async def dispatch(self, request: Request, call_next):
        import time

        # Usa cache per evitare query Redis a ogni richiesta
        now = time.time()
        if now - self._cache_ts > self._cache_ttl:
            flag = await self.redis.get(KILL_SWITCH_KEY)
            self._cached_state = flag is not None and flag.decode() == "1"
            self._cache_ts = now

        # Se kill switch attivo e richiesta verso endpoint AI: devia a OpenAI
        if self._cached_state and request.url.path.startswith("/v1/chat/"):
            return await self._forward_to_openai(request)

        return await call_next(request)

    async def _forward_to_openai(self, request: Request) -> Response:
        """
        Inoltra la richiesta a OpenAI mantenendo la stessa struttura request/response.
        La chiave API è SOLO in variabile d'ambiente server-side, mai nel client.
        """
        openai_key = os.environ.get("OPENAI_FALLBACK_API_KEY")
        if not openai_key:
            # Nessuna chiave configurata: fallback non disponibile, restituisce 503
            return Response(
                content='{"error":"fallback_unavailable","message":"Self-hosted LLM non disponibile e nessun fallback configurato"}',
                status_code=503,
                media_type="application/json"
            )

        body = await request.body()

        async with httpx.AsyncClient(timeout=60.0) as client:
            resp = await client.post(
                OPENAI_FALLBACK_URL,
                content=body,
                headers={
                    "Authorization": f"Bearer {openai_key}",
                    "Content-Type": "application/json"
                }
            )

        # Aggiunge header per tracciare che la risposta viene dal fallback
        headers = dict(resp.headers)
        headers["X-Llm-Provider"] = "openai-fallback"
        headers["X-Kill-Switch-Active"] = "1"

        return Response(
            content=resp.content,
            status_code=resp.status_code,
            headers=headers,
            media_type="application/json"
        )


# API interna per attivare/disattivare il kill switch (richiede autenticazione admin)
# POST /internal/kill-switch/activate
# POST /internal/kill-switch/deactivate
async def activate_kill_switch(redis_client: aioredis.Redis, reason: str):
    """Attiva il kill switch e logga il motivo con timestamp."""
    import time
    await redis_client.set(KILL_SWITCH_KEY, "1")
    await redis_client.set(
        "llm:kill_switch:reason",
        f"{time.strftime('%Y-%m-%dT%H:%M:%SZ')} | {reason}"
    )

async def deactivate_kill_switch(redis_client: aioredis.Redis):
    """Disattiva il kill switch e riporta il traffico verso self-hosted."""
    await redis_client.delete(KILL_SWITCH_KEY)
    await redis_client.delete("llm:kill_switch:reason")

Section 7 - Stress Test: Verify System Reliability

Before going into production, you need to verify that rate limits, circuit breakers, and cost caps function effectively under real load. The following script simulates 50 concurrent users sending requests in a loop, checking that the system responds with correct status codes (200 for requests within the limit, 429 for those beyond the threshold, 503 when the circuit breaker is open).

#!/bin/bash
# stress_test_llm.sh - Script di stress test per production safety LLM
# Uso: ./stress_test_llm.sh <BASE_URL> <JWT_TOKEN> <CONCURRENT_USERS>

BASE_URL="${1:-http://localhost:8080}"
JWT_TOKEN="${2:-your-jwt-token-here}"
CONCURRENT_USERS="${3:-50}"
DURATION_SECONDS=60
REQUEST_INTERVAL=0.1  # 10 req/sec per utente = 500 req/sec totali

# Contatori risultati
count_200=0
count_429=0
count_503=0
count_other=0

echo "=== LLM Production Safety Stress Test ==="
echo "Target:      $BASE_URL"
echo "Concurrency: $CONCURRENT_USERS utenti"
echo "Duration:    ${DURATION_SECONDS}s"
echo "Rate:        $(echo "$CONCURRENT_USERS / $REQUEST_INTERVAL" | bc) req/sec totali"
echo ""

# Funzione per un singolo utente concorrente
run_user() {
    local user_id=$1
    local end_time=$((SECONDS + DURATION_SECONDS))
    local local_200=0 local_429=0 local_503=0

    while [ $SECONDS -lt $end_time ]; do
        response=$(curl -s -o /dev/null -w "%{http_code}" \
            -X POST "$BASE_URL/api/v1/ai/chat" \
            -H "Authorization: Bearer $JWT_TOKEN" \
            -H "Content-Type: application/json" \
            -d "{\"model\":\"llama3.1:8b\",\"message\":\"Cosa è il rate limiting?\",\"user_id\":\"test-user-$user_id\"}" \
            --max-time 10)

        case $response in
            200) ((local_200++)) ;;
            429) ((local_429++)) ;;
            503) ((local_503++)) ;;
        esac

        sleep $REQUEST_INTERVAL
    done

    # Scrive risultati in file temporaneo (thread-safe via append)
    echo "$local_200 $local_429 $local_503" >> /tmp/stress_results.$$
}

# Avvia utenti concorrenti in background
rm -f /tmp/stress_results.$$
for i in $(seq 1 $CONCURRENT_USERS); do
    run_user $i &
done

# Aspetta completamento
wait

# Aggrega risultati
while IFS=' ' read -r c200 c429 c503; do
    ((count_200 += c200))
    ((count_429 += c429))
    ((count_503 += c503))
done < /tmp/stress_results.$$

total=$((count_200 + count_429 + count_503))
echo ""
echo "=== Risultati Stress Test ==="
echo "Totale richieste:     $total"
echo "200 OK (entro limite): $count_200 ($(( count_200 * 100 / total ))%)"
echo "429 Rate Limited:      $count_429 ($(( count_429 * 100 / total ))%)"
echo "503 Circuit Open:      $count_503 ($(( count_503 * 100 / total ))%)"
echo ""
echo "Verifica attesa:"
echo "  - 429% dovrebbe essere ~60-80% con $CONCURRENT_USERS utenti e rate limit FREE=10req/min"
echo "  - 503% dovrebbe scattare se GPU utilization supera 90% durante il test"
echo "  - 200% dovrebbe essere stabile e non degradare nel tempo"

rm -f /tmp/stress_results.$$

Interpret the results as follows: if the percentage of 429 is zero even with 50 concurrent users, the rate limit is not working. If the percentage of 503 appears immediately without the GPU being under stress, the circuit breaker has overly conservative thresholds. The target is 429 high (the defense system), 503 low (the opening system that only opens in real emergency situations), and 200 stable over time (users within the threshold always receive a response).

Frequently Asked Questions

What is the difference between rate limiting and circuit breakers in a self-hosted LLM?

Rate limiting controls the number of requests a single user or endpoint can make within a given time interval, returning 429 when it exceeds the threshold. The circuit breaker monitors the health of the LLM service itself (GPU utilization, error rate, latency) and opens the circuit refusing all requests when the system is under stress, regardless of the requester. The two patterns complement each other: rate limiting protects against individual abusive requests, while the circuit breaker protects the infrastructure from cascading failures.

How to calculate a realistic cost cap for a session?

Calculate the cost per token by dividing the hourly hardware cost by throughput in tokens per hour: GPU A100 at €2/hour with vLLM at 800 tokens/second = €2 / (800 × 3600) = €0.00000069 per token. Multiply by desired threshold: 50,000 tokens (Pro tier) = €0.035 per session. This approach makes the cost cap independent of cloud prices and anchored to real infrastructure costs.

What happens if the GPU reaches 95% utilization?

With GPU utilization at 95%, new requests are accommodated without finding free computational capacity: the TTFT latency starts to grow non-linearly, until it exceeds client timeouts. The circuit breaker should open first, typically at >85-90% sustained for 30 seconds, to prevent the queue from growing indefinitely. Incoming requests receive an immediate 503 instead of waiting indefinitely.

Queue or Reject-Fast: When to Choose?

Choose the queue when variable latency is tolerated by usage cases: batch processing at night, asynchronous summarization, analysis pipeline. Choose reject-fast when you have strict latency SLA (interactive chatbot, real-time copilot) because users prefer immediate errors over 30-second delays. In practice, many systems use both: a short queue (10-20 elements) to absorb temporary spikes, then reject-fast when the queue is full.

How to implement a fallback to a cloud provider without exposing the API key in the frontend?

The fallback must always transit through the backend (Spring Boot or FastAPI), never from the frontend. The middleware intercepts requests, and if the circuit is open, it redirects the call to the external provider using the API key stored as a server-side variable. The client receives the same response structure without knowing which provider served the request. The API key never appears in the JavaScript bundle or public access logs.

What metrics are essential for Prometheus to ensure an SLA of 99%?

The critical metrics are: vllm_time_to_first_token_seconds (TTF Histogram), vllm_e2e_request_latency_seconds Latency end-to-end. vllm_num_requests_waiting I'm ready to translate. Please go ahead and provide the Italian text. vllm_gpu_cache_usage_perc ( Cache overflow ), and a custom counter for tokens per user per session. Alerts must trigger when histogram_quantile(0.99) Surpasses the threshold SLA for at least 2 minutes, avoiding false positives from short-term moments.

Can a Spring Boot backend with Bucket4j handle multiple AI models with different rates?

Yes. The approach shown in the article uses two separate buckets: one per-user (with capacity dependent on tier) and one per-model (with global capacity to avoid GPU monopolization). Bucket4j on distributed Redis ensures that counters are consistent even with multiple instances of Spring Boot in load balancing. The Redis key bucket:model:llama3.1:70b It is shared among all instances and uses CAS (Compare-And-Swap) for atomic updates without explicit locks.

Conclusion

An LLM self-hosted without guardrails is a time bomb. The four protections of this episode form a deep defense system: the rate limit Bucket4j stops abuse before it reaches vLLM; the GPU-aware circuit breaker opens proactively when hardware is under stress; the cost cap Redis prevents a single session from exhausting the budget; Prometheus alerts on p99 provide immediate visibility of SLA violations before they become incidents.

The choice between queue and reject-fast is strategic and not technical: it depends on who uses the system and what they expect. For an interactive chatbot, an immediate error is always preferable to a long wait. For a batch analysis pipeline, the queue is essential. The kill switch with fallback cloud is the final security network, to be activated only in emergency situations and never as the primary path - privacy and cost control remain at the center of the strategy.

Next episode: EP05 - Monitoring self-hosted LLM: GPU utilization, token latency, cost tracking in real-timeWe will build a complete Grafana dashboard, Prometheus exporter for vLLM and Ollama, and a user-tracking system of costs visible in real-time.