Circuit breaker LLM: Resilience4j + smart recovery + half-open probing

Circuit breaker LLM: Resilience4j + smart recovery + half-open probing

The circuit breaker is born from electrical mechanics: when the current exceeds the safety threshold, the fuse opens and interrupts the circuit before the installation burns. In software, the same pattern protects distributed systems from cascade failure - when a service goes down it drags all those that call it with it. For an LLM in production, however, the classic circuit breaker is too coarse. It opens on a global percentage of errors, not distinguishing between a slow model and an unreachable provider, not knowing that a 50k token query has more likelihood of timing out than one from 200 tokens. The result: false positives opening the circuit when the provider is healthy, or false negatives leaving it closed while silently degrading.

This episode builds a smart circuit breaker for LLM, starting from the Resilience4j configuration in the backend Spring Boot that manages calls to the ml-service FastAPI with Ollama on-prem. The multi-dimensional structure for provider × model × region avoids false positives. The half-open probing with synthetic canary requests protects real users during recovery. The slow call threshold opens the circuit even when the model responds - but in 15 seconds instead of 2. Everything is exported to Prometheus with AlertManager alerts ready for production.

backend portfolio (Spring Boot 3.4 + Resilience4j 2.x) calls to ml-service were originally without circuit breakers. The first documented incident was Ollama saturated, the ml-service in timeout on every request, Spring Boot accumulating threads waiting until exhaustion of the connection pool. A total of fifteen minutes of downtime for a problem that would have been contained by a circuit breaker in sixty seconds. From that day forward, each cross-service call had its own CB configured.

Section 1 - Resilience4j setup: dependencies and configuration for Spring Boot 3

Resilience4j is the de facto resilience library for Java/Spring Boot since 2020, when Hystrix from Netflix went into maintenance. The version 2.x - compatible with Spring Boot 3 and Java 21 - introduces improvements to Micrometer integration that make Prometheus export practically zero-config. The necessary dependencies for circuit breakers on HTTP WebClient calls to ml-service are three: the core Spring Boot 3 module, AOP support for annotations, and the actuator for metrics.

<!-- pom.xml - dipendenze Resilience4j per Spring Boot 3 -->
<dependencies>
    <!-- Circuit breaker + retry + bulkhead + rate limiter -->
    <dependency>
        <groupId>io.github.resilience4j</groupId>
        <artifactId>resilience4j-spring-boot3</artifactId>
        <version>2.2.0</version>
    </dependency>

    <!-- AOP obbligatorio per le annotazioni @CircuitBreaker -->
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-aop</artifactId>
    </dependency>

    <!-- Actuator + Micrometer per export Prometheus -->
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-actuator</artifactId>
    </dependency>
    <dependency>
        <groupId>io.micrometer</groupId>
        <artifactId>micrometer-registry-prometheus</artifactId>
    </dependency>
</dependencies>

With dependencies in place, the YAML configuration defines instances of the circuit breaker. For our backend portfolio we use two separate instances: mlServiceOllama for calls to the AI APIs of the ml-service (high latency, frequent timeouts and mlServiceHealth for health checks (low latency, unexpected failures). Separate instances mean a separate state machine: a timeout on a complex query does not open the health check circuit.

# application.yml - configurazione Resilience4j per chiamate ml-service LLM
resilience4j:
  circuitbreaker:
    configs:
      # Configurazione base condivisa
      default:
        register-health-indicator: true
        event-consumer-buffer-size: 20

      # Configurazione ottimizzata per LLM con alta latenza
      llmAware:
        sliding-window-type: COUNT_BASED
        sliding-window-size: 20              # Valuta sulle ultime 20 call
        minimum-number-of-calls: 10          # Almeno 10 call prima di decidere
        failure-rate-threshold: 30           # Apri se 30%+ delle call fallisce
        slow-call-rate-threshold: 50         # Apri se 50%+ è più lenta del threshold
        slow-call-duration-threshold: 10s    # "Lenta" = oltre 10 secondi
        wait-duration-in-open-state: 30s     # Rimani OPEN 30s prima di provare HALF_OPEN
        permitted-number-of-calls-in-half-open-state: 3
        automatic-transition-from-open-to-half-open-enabled: true
        record-exceptions:
          - java.io.IOException
          - java.util.concurrent.TimeoutException
          - org.springframework.web.reactive.function.client.WebClientResponseException$ServiceUnavailable
          - org.springframework.web.reactive.function.client.WebClientResponseException$TooManyRequests
        ignore-exceptions:
          # Non apre il CB per errori di validazione input - sono errori del chiamante
          - org.springframework.web.reactive.function.client.WebClientResponseException$BadRequest
          - org.springframework.web.reactive.function.client.WebClientResponseException$UnprocessableEntity

    instances:
      # Chiamate AI verso ml-service (chat, recommend, summarize, explain-code)
      mlServiceOllama:
        base-config: llmAware
        # Soglia più bassa per endpoint AI critici
        failure-rate-threshold: 25
        slow-call-duration-threshold: 15s    # Ollama on-prem può essere lento su query lunghe

      # Health check ml-service - soglie diverse, non mescolare con AI call
      mlServiceHealth:
        base-config: default
        sliding-window-size: 5
        minimum-number-of-calls: 3
        failure-rate-threshold: 60           # Health check fallisce più spesso: soglia più alta
        slow-call-duration-threshold: 2s     # Health check deve essere veloce

# Actuator: esponi endpoint circuitbreakers per Prometheus
management:
  endpoints:
    web:
      exposure:
        include: health,metrics,prometheus,circuitbreakers,circuitbreakerevents
  health:
    circuitbreakers:
      enabled: true
  metrics:
    export:
      prometheus:
        enabled: true

Section 2 - Implementation of Java: @CircuitBreaker on WebClient towards ml-service

With the YAML configuration in place, the circuit breaker is activated through annotation @CircuitBreaker on the method that calls the ml-service. In the backend portfolio, MlServiceClient It is the component that wraps all HTTP calls to FastAPI. Each exposed method has its own callback named and its fallback method - a method with the same signature plus an additional parameter. Throwable That receives an exception and returns a degraded response.

// MlServiceClient.java - WebClient con @CircuitBreaker per chiamate Ollama
package dev.federicocalo.portfolio.infrastructure.ai;

import io.github.resilience4j.circuitbreaker.annotation.CircuitBreaker;
import org.springframework.stereotype.Component;
import org.springframework.web.reactive.function.client.WebClient;
import reactor.core.publisher.Mono;

import java.time.Duration;
import java.util.Map;
import java.util.logging.Logger;

@Component
public class MlServiceClient {

    private static final Logger log = Logger.getLogger(MlServiceClient.class.getName());

    private final WebClient webClient;

    public MlServiceClient(WebClient.Builder builder) {
        this.webClient = builder
            .baseUrl("http://localhost:8000")  // ml-service FastAPI locale
            .build();
    }

    /**
     * Chiamata al endpoint /internal/ai/chat con circuit breaker.
     * Se il CB è OPEN o la call fallisce, viene invocato chatFallback().
     */
    @CircuitBreaker(name = "mlServiceOllama", fallbackMethod = "chatFallback")
    public Mono<AiChatResponse> chat(AiChatRequest request) {
        return webClient.post()
            .uri("/internal/ai/chat")
            .bodyValue(request)
            .retrieve()
            .bodyToMono(AiChatResponse.class)
            .timeout(Duration.ofSeconds(20))  // Timeout WebClient sotto il slow-call-duration-threshold
            .doOnError(e -> log.warning("ml-service chat error: " + e.getMessage()));
    }

    /**
     * Fallback per chat(): risposta degradata con flag per UX.
     * Stessa firma di chat() + Throwable come ultimo parametro.
     */
    private Mono<AiChatResponse> chatFallback(AiChatRequest request, Throwable ex) {
        log.warning("Circuit breaker attivo per mlServiceOllama. Causa: " + ex.getClass().getSimpleName());
        // Risposta degradata: l'utente vede un messaggio esplicativo invece di un errore
        return Mono.just(AiChatResponse.degraded(
            "Il servizio AI è temporaneamente non disponibile. Riprova tra qualche minuto.",
            ex.getMessage()
        ));
    }

    /**
     * Chiamata a /internal/ai/recommend - endpoint più leggero, stesso CB.
     */
    @CircuitBreaker(name = "mlServiceOllama", fallbackMethod = "recommendFallback")
    public Mono<AiRecommendResponse> recommend(String articleSlug, int limit) {
        return webClient.get()
            .uri(uriBuilder -> uriBuilder
                .path("/internal/ai/recommend")
                .queryParam("slug", articleSlug)
                .queryParam("limit", limit)
                .build())
            .retrieve()
            .bodyToMono(AiRecommendResponse.class)
            .timeout(Duration.ofSeconds(8));
    }

    private Mono<AiRecommendResponse> recommendFallback(String articleSlug, int limit, Throwable ex) {
        // Per i recommend usiamo articoli correlati statici dal database come fallback
        return Mono.just(AiRecommendResponse.staticFallback(articleSlug));
    }

    /**
     * Health check verso ml-service - CB separato, soglie diverse.
     */
    @CircuitBreaker(name = "mlServiceHealth", fallbackMethod = "healthFallback")
    public Mono<Map<String, String>> health() {
        return webClient.get()
            .uri("/health")
            .retrieve()
            .bodyToMono(Map.class)
            .timeout(Duration.ofSeconds(3));
    }

    private Mono<Map<String, String>> healthFallback(Throwable ex) {
        return Mono.just(Map.of("status", "degraded", "reason", ex.getMessage()));
    }
}

The key point of this design: CB mlServiceOllama is shared among chat() e recommend()but the CB mlServiceHealth It is isolated. If Ollama is saturated and both chat and recommend fail, the circuit opens for both - correct because they share the same provider. The health check instead remains operational and continues to signal the real state of the ml-service on Prometheus.

Section 3 - Half-open probing: synthetic canary instead of real traffic

The transition from OPEN to HALF_OPEN is the most delicate moment in the lifecycle of a Circuit Breaker LLM. The standard CB of Resilience4j allows permitted-number-of-calls-in-half-open-state Requests for real testing of recovery. The problem is that those 3 requests could be from premium users, query of 50k tokens, or critical operations. If the provider is still unstable, you're degrading it for nothing. The solution is the probing canary: a synthetic, lightweight, idempotent request that verifies availability without impact on the user.

# ml_service/app/providers/circuit_breaker.py
# Circuit breaker Python puro con half-open probing canary per Ollama
import asyncio
import time
import logging
from dataclasses import dataclass, field
from enum import Enum
from typing import Callable, Any, Optional

logger = logging.getLogger(__name__)


class CBState(Enum):
    CLOSED = "closed"        # Operativo: tutto il traffico passa
    OPEN = "open"            # Aperto: tutto il traffico bloccato, solo canary
    HALF_OPEN = "half_open"  # Recupero: canary passato, test con traffico limitato


@dataclass
class CircuitBreakerConfig:
    failure_threshold: int = 5       # N errori consecutivi per aprire
    failure_rate: float = 0.30       # 30% failure rate → apri
    slow_call_threshold_s: float = 10.0  # Call oltre 10s = slow call
    slow_call_rate: float = 0.50     # 50% slow calls → apri
    recovery_timeout_s: float = 30.0     # Attesa in OPEN prima di HALF_OPEN
    window_size: int = 20            # Finestra di valutazione (ultime N call)
    half_open_max_calls: int = 3     # Max call in HALF_OPEN prima di decidere


@dataclass
class CallRecord:
    success: bool
    duration_s: float
    timestamp: float = field(default_factory=time.monotonic)


class LLMCircuitBreaker:
    """
    Circuit breaker ottimizzato per chiamate LLM on-prem (Ollama).
    Implementa failure rate + slow call rate con half-open canary probing.
    """

    def __init__(self, name: str, config: CircuitBreakerConfig = None):
        self.name = name
        self.config = config or CircuitBreakerConfig()
        self._state = CBState.CLOSED
        self._call_history: list[CallRecord] = []
        self._opened_at: Optional[float] = None
        self._half_open_calls = 0
        self._lock = asyncio.Lock()

    @property
    def state(self) -> CBState:
        return self._state

    async def call(self, fn: Callable, *args, **kwargs) -> Any:
        """
        Wrappa una chiamata LLM con circuit breaker.
        Lancia CircuitOpenError se il CB è OPEN e il canary non è passato.
        """
        async with self._lock:
            if self._state == CBState.OPEN:
                if not await self._should_attempt_recovery():
                    raise CircuitOpenError(f"Circuit breaker {self.name} è OPEN")
                # Transizione a HALF_OPEN: prima esegui canary
                await self._run_canary()
                self._state = CBState.HALF_OPEN
                self._half_open_calls = 0
                logger.info(f"[CB:{self.name}] Transizione OPEN → HALF_OPEN dopo canary OK")

            if self._state == CBState.HALF_OPEN:
                if self._half_open_calls >= self.config.half_open_max_calls:
                    # Non siamo ancora sicuri: blocca il traffico extra
                    raise CircuitOpenError(f"Circuit breaker {self.name} in HALF_OPEN: limite raggiunto")
                self._half_open_calls += 1

        # Esegui la call reale fuori dal lock per non bloccare concorrenti
        start = time.monotonic()
        try:
            result = await fn(*args, **kwargs)
            duration = time.monotonic() - start
            await self._record_success(duration)
            return result
        except Exception as e:
            duration = time.monotonic() - start
            await self._record_failure(duration)
            raise

    async def _run_canary(self):
        """
        Probe sintetico: chiama Ollama con un prompt minimale ('Rispondi OK').
        Se fallisce, rimane OPEN. Se passa, procede con HALF_OPEN.
        """
        try:
            # Import inline per evitare circular dependency
            from app.providers.ollama import OllamaProvider
            provider = OllamaProvider.get_instance()
            # Prompt minimale, deterministico, bassa latenza attesa
            response = await asyncio.wait_for(
                provider.generate(
                    prompt="Rispondi con una sola parola: OK",
                    model="llama3.1:8b",
                    max_tokens=5
                ),
                timeout=5.0  # Canary deve rispondere in 5s o consideriamo OPEN ancora
            )
            if "ok" not in response.lower():
                raise ValueError(f"Canary response inattesa: {response!r}")
            logger.info(f"[CB:{self.name}] Canary probe OK: '{response.strip()}'")
        except Exception as e:
            logger.warning(f"[CB:{self.name}] Canary probe FALLITO: {e}. CB rimane OPEN")
            raise CircuitOpenError(f"Canary probe fallito per {self.name}: {e}") from e

    async def _record_success(self, duration_s: float):
        async with self._lock:
            record = CallRecord(success=True, duration_s=duration_s)
            self._call_history.append(record)
            self._trim_history()

            if self._state == CBState.HALF_OPEN and self._half_open_calls >= self.config.half_open_max_calls:
                self._state = CBState.CLOSED
                self._opened_at = None
                logger.info(f"[CB:{self.name}] Transizione HALF_OPEN → CLOSED dopo {self._half_open_calls} call ok")

    async def _record_failure(self, duration_s: float):
        async with self._lock:
            record = CallRecord(success=False, duration_s=duration_s)
            self._call_history.append(record)
            self._trim_history()

            if self._state == CBState.HALF_OPEN:
                # Una failure in HALF_OPEN riapre subito
                self._state = CBState.OPEN
                self._opened_at = time.monotonic()
                logger.warning(f"[CB:{self.name}] Failure in HALF_OPEN → OPEN")
                return

            if self._should_open():
                self._state = CBState.OPEN
                self._opened_at = time.monotonic()
                logger.warning(f"[CB:{self.name}] Soglia superata → OPEN")

    def _should_open(self) -> bool:
        """Valuta se failure rate o slow call rate superano la soglia."""
        if len(self._call_history) < self.config.window_size // 2:
            return False  # Dati insufficienti

        window = self._call_history[-self.config.window_size:]
        total = len(window)
        failures = sum(1 for r in window if not r.success)
        slow_calls = sum(1 for r in window if r.duration_s > self.config.slow_call_threshold_s)

        failure_rate = failures / total
        slow_rate = slow_calls / total

        if failure_rate >= self.config.failure_rate:
            logger.warning(f"[CB:{self.name}] Failure rate {failure_rate:.1%} >= {self.config.failure_rate:.1%}")
            return True
        if slow_rate >= self.config.slow_call_rate:
            logger.warning(f"[CB:{self.name}] Slow call rate {slow_rate:.1%} >= {self.config.slow_call_rate:.1%}")
            return True
        return False

    async def _should_attempt_recovery(self) -> bool:
        elapsed = time.monotonic() - (self._opened_at or 0)
        return elapsed >= self.config.recovery_timeout_s

    def _trim_history(self):
        if len(self._call_history) > self.config.window_size * 2:
            self._call_history = self._call_history[-self.config.window_size:]


class CircuitOpenError(Exception):
    """Lanciato quando il circuit breaker è OPEN e non permette il passaggio."""
    pass

Section 4 - Smart Recovery: Exponential Backoff with Jitter

The circuit breaker manages the opening and closing of the circuit, but not the retry strategy in situations where it makes sense to try again. A closed LLM with sporadic errors should not open the circuit if transient errors are present and an immediate retry resolves them. The optimal combination is: retry with exponential backoff + jitter before counting failures towards the circuit breaker. Resilience4j handles this with the combination @Retry + @CircuitBreaker The retry occurs first, and only if all retries fail does the CB register the failure.

# ml_service/app/providers/retry_policy.py
# Exponential backoff con jitter per chiamate Ollama transitoriamente fallimentari
import asyncio
import random
import logging
from typing import Callable, Any, Type

logger = logging.getLogger(__name__)


async def with_exponential_backoff(
    fn: Callable,
    *args,
    max_attempts: int = 3,
    base_delay_s: float = 1.0,
    max_delay_s: float = 30.0,
    jitter: bool = True,
    retryable_exceptions: tuple[Type[Exception], ...] = (IOError, TimeoutError),
    **kwargs
) -> Any:
    """
    Esegue fn con retry exponential backoff + jitter.
    Lancia l'ultima eccezione se tutti i tentativi falliscono.

    Delay formula: min(base * 2^attempt + jitter, max_delay)
    Jitter range: +/- 20% del delay calcolato
    """
    last_exception = None

    for attempt in range(max_attempts):
        try:
            return await fn(*args, **kwargs)

        except retryable_exceptions as e:
            last_exception = e
            if attempt == max_attempts - 1:
                # Ultimo tentativo fallito - propaga al circuit breaker
                logger.error(f"Tutti {max_attempts} tentativi falliti. Ultimo errore: {e}")
                raise

            # Calcola delay con exponential backoff
            delay = min(base_delay_s * (2 ** attempt), max_delay_s)
            if jitter:
                # Jitter ±20% per evitare thundering herd (tutti i client retryin insieme)
                jitter_amount = delay * 0.2
                delay = delay + random.uniform(-jitter_amount, jitter_amount)

            logger.warning(
                f"Tentativo {attempt + 1}/{max_attempts} fallito ({e}). "
                f"Retry tra {delay:.1f}s"
            )
            await asyncio.sleep(delay)

        except Exception as e:
            # Eccezione non retryable: propaga immediatamente senza retry
            logger.error(f"Errore non retryable: {type(e).__name__}: {e}")
            raise

    raise last_exception  # Safety net - non dovrebbe mai arrivare qui

Jitter is crucial in an LLM context with many competing users. Without jitter, if 50 requests fail together, all retries occur exactly 2 seconds later - flooding the provider again at the exact moment it could have partially recovered. With jitter ±20%, the 50 retries are spread out over a 400ms window, reducing the peak load by 60-70%.

Section 5 - Prometheus metrics for state circuit breakers

Without observation, the circuit breaker is a black box: you know it opens and closes, but not when, for how long, and at what frequency. Resilience4j automatically exports necessary metrics via Micrometer. The most important metric for production monitoring is resilience4j_circuitbreaker_stateA gauge that is worth 0 (CLOSED), 1 (OPEN), or 2 (HALF_OPEN). Any value other than 0 for more than 60 seconds is a problem signal that requires attention.

# prometheus/rules/llm_circuit_breaker.yml
# Regole di alerting per circuit breaker LLM - AlertManager
groups:
  - name: llm_circuit_breaker
    interval: 15s
    rules:

      # Alert quando il CB è OPEN per più di 60 secondi
      - alert: LLMCircuitBreakerOpen
        expr: resilience4j_circuitbreaker_state{name="mlServiceOllama"} == 1
        for: 60s
        labels:
          severity: critical
          team: ai-platform
        annotations:
          summary: "Circuit breaker mlServiceOllama è OPEN"
          description: |
            Il circuit breaker per le chiamate AI è aperto da più di 60 secondi.
            Failure rate corrente: {{ $value }}
            Azione: verifica stato Ollama su VPS (systemctl status ollama)

      # Alert quando il CB è in HALF_OPEN - indica recovery in corso
      - alert: LLMCircuitBreakerHalfOpen
        expr: resilience4j_circuitbreaker_state{name="mlServiceOllama"} == 2
        for: 120s
        labels:
          severity: warning
          team: ai-platform
        annotations:
          summary: "Circuit breaker mlServiceOllama bloccato in HALF_OPEN"
          description: |
            Il CB è in HALF_OPEN da più di 2 minuti. Il recovery non completa.
            Possibile causa: Ollama risponde al canary ma fallisce sul traffico reale.

      # Alert su failure rate elevato anche con CB CLOSED
      - alert: LLMHighFailureRate
        expr: |
          resilience4j_circuitbreaker_failure_rate{name="mlServiceOllama"} > 20
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Failure rate LLM > 20% (CB ancora CLOSED)"
          description: |
            Failure rate: {{ printf "%.1f" $value }}%
            Il CB non si è ancora aperto ma la degradazione è in atto.

      # Alert su slow call rate - segnale di saturazione Ollama
      - alert: LLMHighSlowCallRate
        expr: |
          resilience4j_circuitbreaker_slow_call_rate{name="mlServiceOllama"} > 40
        for: 3m
        labels:
          severity: warning
        annotations:
          summary: "Slow call rate LLM > 40% - Ollama sotto pressione"
          description: |
            Il {{ printf "%.1f" $value }}% delle call supera la soglia di 10s.
            Azione: verifica load medio Ollama, considera riduzione concurrency.

For the Python machine learning service, the same metrics are exposed through prometheus_client Updating manually the gauges inside the circuit breaker's internal state. The key is to update the gauge every time the CB's state changes, not through polling - so Prometheus always collects the current value at the moment of scraping.

# ml_service/app/providers/circuit_breaker_metrics.py
# Esportazione metriche Prometheus per il circuit breaker Python
from prometheus_client import Gauge, Counter, Histogram
from .circuit_breaker import CBState

# Gauge di stato: 0=CLOSED, 1=OPEN, 2=HALF_OPEN (allineato a Resilience4j)
cb_state_gauge = Gauge(
    "llm_circuit_breaker_state",
    "Stato del circuit breaker LLM (0=CLOSED, 1=OPEN, 2=HALF_OPEN)",
    labelnames=["name"]
)

# Contatori per le transizioni di stato
cb_transitions = Counter(
    "llm_circuit_breaker_transitions_total",
    "Numero di transizioni di stato del circuit breaker",
    labelnames=["name", "from_state", "to_state"]
)

# Histogram per la durata delle call (permette p50, p95, p99)
cb_call_duration = Histogram(
    "llm_circuit_breaker_call_duration_seconds",
    "Durata delle call LLM attraverso il circuit breaker",
    labelnames=["name", "outcome"],  # outcome: success | failure | slow
    buckets=[1, 2, 5, 10, 15, 20, 30, 60]
)

# Contatore per le call bloccate dal CB OPEN
cb_rejected_calls = Counter(
    "llm_circuit_breaker_rejected_calls_total",
    "Call rifiutate dal circuit breaker quando OPEN",
    labelnames=["name"]
)


def update_state_metric(cb_name: str, new_state: CBState):
    """Chiama ogni volta che lo stato del CB cambia."""
    state_value = {
        CBState.CLOSED: 0,
        CBState.OPEN: 1,
        CBState.HALF_OPEN: 2
    }[new_state]
    cb_state_gauge.labels(name=cb_name).set(state_value)

Section 6 - Reference Tables: States and Tuning Thresholds

Before arriving at empirical tuning, it is useful to have a clear reference of the circuit breaker states and the expected behavior in each scenario LLM. The following table describes the three states with traffic flow and suggested actions.

The tuning of the threshold is the most critical part - and is often overlooked. The default Resilience4j settings (50% failure rate, 100ms call window) are designed for standard HTTP services with latency of a few milliseconds. For an on-prem LLM with latency of 2-15 seconds and baseline failure rate of 3-8%, these defaults are too high and too slow. The following table provides empirically tested starting values for various use cases LLM.

Section 7 - FastAPI middleware: integration into the ML service

FastAPI ML service, the circuit breaker Ollama operates as middleware at the provider level, not at the HTTP endpoint level. The distinction is important: we don't want to block all requests to the ML service when Ollama is down - deterministic endpoints (health, tool assist without AI) must continue to function. Only requests that pass through OllamaProvider.generate() They go through the CB.

# ml_service/app/providers/ollama.py - provider con CB integrato
import asyncio
import logging
from app.providers.circuit_breaker import LLMCircuitBreaker, CircuitOpenError, CircuitBreakerConfig

logger = logging.getLogger(__name__)

# Istanza globale del circuit breaker - vive per tutta la durata dell'app
_ollama_cb = LLMCircuitBreaker(
    name="ollama_llama3",
    config=CircuitBreakerConfig(
        failure_threshold=5,
        failure_rate=0.25,          # 25% failure rate
        slow_call_threshold_s=12.0,  # 12s = slow per chat interattiva
        slow_call_rate=0.50,
        recovery_timeout_s=30.0,
        window_size=20,
        half_open_max_calls=3
    )
)


class OllamaProvider:
    """Provider Ollama con circuit breaker e retry integrati."""

    _instance: "OllamaProvider | None" = None

    @classmethod
    def get_instance(cls) -> "OllamaProvider":
        if cls._instance is None:
            cls._instance = cls()
        return cls._instance

    async def generate(self, prompt: str, model: str = "llama3.1:8b", **kwargs) -> str:
        """
        Genera risposta Ollama con circuit breaker.
        Lancia CircuitOpenError se il CB è OPEN.
        Il caller (endpoint FastAPI) gestisce il fallback.
        """
        try:
            return await _ollama_cb.call(
                self._raw_generate,
                prompt=prompt,
                model=model,
                **kwargs
            )
        except CircuitOpenError:
            logger.warning(f"Ollama CB OPEN: richiesta bloccata per '{prompt[:50]}...'")
            raise  # Il caller decide il fallback (cache, statico, errore esplicito)

    async def _raw_generate(self, prompt: str, model: str, max_tokens: int = 1024) -> str:
        """Call raw verso Ollama API - nessuna logica di resilienza qui."""
        import httpx
        async with httpx.AsyncClient(timeout=20.0) as client:
            response = await client.post(
                "http://localhost:11434/api/generate",
                json={
                    "model": model,
                    "prompt": prompt,
                    "stream": False,
                    "options": {"num_predict": max_tokens}
                }
            )
            response.raise_for_status()
            data = response.json()
            return data.get("response", "")

The endpoint FastAPI that uses the provider manages CircuitOpenError with the strategy of graceful degradation described in EP03: response from Redis cache if available, otherwise explanatory message with flags. ai_degraded: true response body to allow the Angular frontend to display the correct disclaimer to the user.

Frequently Asked Questions

Why a generic circuit breaker is not enough for a production LLM?

A generic CB treats all errors equally and opens on a global threshold. For an LLM you need to distinguish by provider specific, model, and query type. A multi-dimensional CB avoids costly false positives: opening the circuit when the provider is healthy causes unnecessary downtime, while keeping it closed during silent degradation causes degraded UX without visibility.

What is the half-open probing canary and why is it superior to the standard?

HALF_OPEN standard state, CB allows real requests to pass through for testing recovery. With an LLM, those real requests can be costly queries of premium users. The canary uses instead synthetic light requests - a brief prompt like "Respond with OK" - to check availability without consuming real resources or degrading user experience during recovery.

What is the difference between failure rate threshold and slow call rate threshold?

The failure rate threshold opens the circuit when a percentage of calls returns an error. The slow call rate threshold opens the circuit when a percentage of calls exceeds the slow call duration threshold - even without explicit errors. For LLMs, this distinction is crucial: a model under load can respond correctly but degrade UX in 15-30 seconds without ever throwing an exception.

How to choose the right failure rate threshold for Ollama on-prem?

Measure the failure rate baseline over 7 days: if it's below 5%, set the threshold to 25-30%. Never use the default of 50% without measuring first: you risk opening too late or never. For Ollama on VPS with 8GB VRAM and model llama3.1:8bThe values tested in the portfolio are: failure-rate 25%, slow-call-duration 12s, slow-call-rate 50%.

What does Resilience4j expose on Prometheus to monitor circuit breakers?

The main metrics are: resilience4j_circuitbreaker_state (0 = CLOSED, 1 = OPEN, 2 = HALF_OPEN) resilience4j_circuitbreaker_failure_rate, resilience4j_circuitbreaker_slow_call_rateCall counters and totals, successes, failures, and unauthorized calls. The state gauge is the most useful for alerts: any value != 0 must trigger a notification within more than 60 seconds.

Can I use a circuit breaker directly with FastAPI without Spring Boot?

Yes. For the ml-service you can use pybreaker Alternatively, a pure Python implementation like the one shown in this article. The pure approach with dataclass and asyncio is preferable for total control: no additional dependency, explicit state machine, direct integration with prometheus_client. The FastAPI middleware wraps every call to the provider LLM and updates the CB state.

How to coordinate the circuit breaker in Spring Boot with that in the ml-service FastAPI?

The architecture of the portfolio uses two independent levels: Resilience4j in Spring Boot manages HTTP calls to the ml-service, while the CB Python in the ml-service manages calls to Ollama. The two levels are separated: you can have the CB towards ml-service CLOSED but the CB towards Ollama OPEN - in that case the ml-service returns a degraded response with flags, Spring Boot receives it and propagates it to the user with an appropriate disclaimer.

Conclusions

The circuit breaker LLM is not an option-it's a production requirement. An LLM on-prem like Ollama has failure modes that a system without CB can't absorb: thread pool saturation in Spring Boot, cascade failures to all users, and the inability to recover automatically without manual restarts. The configuration shown in this episode-Resilience4j with empirically tuned thresholds, multi-dimensional CB per instance, half-open canary probing, Prometheus metrics, and AlertManager-reduces recovery time from minutes to seconds, without impacting users during the cycle OPEN → HALF_OPEN → CLOSED.

Practical takeaways: 1. Measure the failure rate baseline over 7 days first. 2. Do not use default Resilience4j settings for LLM. 3. Always separate CB from AI call and health check separately. 4. Implement canary probing instead of half-open standard to protect real users during recovery. 5. For the ml-service Python, pure implementation with asyncio.Lock is more controllable than external libraries and integrates naturally with prometheus_client.

Next episode: Cached response strategy: when showing cache vs loading We build the complete caching strategy for AI responses, with Redis TTL, invalidation pattern and the decision tree to determine when to show the cache to the user and when to prefer the loading indicator.