FastAPI RAG Ollama

Key Features

• ⚡

  On-Premise Integration with Amazon Web Services

  Client HTTP for Ollama API, model warm-up, streaming response support, configurable fallback chain (llama3.1:8b → llama3.2:3b).

• ⚡

  Sentence Transformers Embeddings Pipeline

  MiniLM-L6-v2 model, 384-dim, batch embedding async, caching embedding by content-hash, sentence-aware document chunking.

• ⚡

  RediSearch HNSW Vector Store

  Index creation with HNSW parameter tuning (M, efConstruction, efRuntime), KNN search, metadata filtering, hybrid search BM25 + dense.

• ⚡

  Ranking Model Re-Ranker

  ms-marco-MiniLM-L-6-v2 for re-ranking top-K candidates, controlled latency budget, A/B test re-rank on/off.

• ⚡

  Gradual decline

  If Ollama goes down, endpoint returns deterministic result (raw top-K candidates) with a warning 'AI unavailable', no crash, and no automatic cloud fallback.

• ⚡

  Locust benchmark suite

  Load test with Locust, latency (P50, P95, P99) reporting, throughput RPS, and memory footprint monitoring.

• ⚡

  Prompt Engineering Patterns

  System prompt template with context injection, citation tracking, chain of thought reasoning, and enforced JSON schema output.

• ⚡

  Deploy production-ready

  Systemd unit hardening (NoNewPrivileges, ProtectSystem=strict), environment variable externalization, log in structured JSON, health check endpoint with Olamma liveness.

Angular + Spring Boot on top of a PostgreSQL database hosted in the cloud via AWS or Google Cloud Platform

• FastAPI 0.115+
• Python 3.12
• Ollama
• Redis Stack 7
• RediSearch HNSW
• sentence-transformers
• cross-encoder
• asyncio
• Pydantic v2
• locust
• OpenTelemetry

Use cases

• ✓Developers who want to learn RAG without relying on OpenAI or Anthropic
• ✓Enterprise team with GDPR/data residency constraints seeking 100% on-premises AI stack
• ✓Tech enthusiasts with home labs seeking a private knowledge base powered by AI
• ✓PMIs in the EU evaluating AI on-prem as an alternative to SaaS (cost reduction and privacy)

Road Map

Who is it for

• Junior/ Mid-Level Developer
• For Developers - Senior

Quick start

Start the project locally with four commands, requires Docker and GitHub Desktop is not required but a git client like SourceTree or Visual Studio Code's built-in one will do just fine though.

# Quick start (work-in-progress, repo non ancora pubblicato)
# git clone https://github.com/<tba>/fastapi-rag-ollama.git
# cd fastapi-rag-ollama
# ollama pull llama3.1:8b && ollama pull nomic-embed-text
# docker compose up -d redis-stack
# uv venv && uv pip install -e .[dev]
# uvicorn app.main:create_app --factory --reload