WIP MITInstructors

FastAPI RAG Ollama

Build a self-hosted RAG microservice with FastAPI, Ollama, and RediSearch HNSW.

Status
WIP
License
MIT
Category
Instructors
Updated
GitHub source

Project status

Description

FastAPI RAG Ollama shows how to build a self-hosted RAG (Retrieval Augmented Generation) microservice without depending on cloud AI providers (OpenAI, Anthropic, Cohere, Google). The pattern is taken directly from the portfolio project (`ml-service/`, running in real production on a Hetzner VPS in Nuremberg), distilled into a progressive 8-week tutorial accessible to mid-level Python developers. The path covers, step by step: local Ollama setup (llama3.1:8b as default, with llama3.2:3b, mistral-nemo:12b and qwen2.5:14b discussed as alternatives with latency and quality trade-offs), embedding generation with sentence-transformers all-MiniLM-L6-v2 (384 dimensions, async batching for throughput), a RediSearch HNSW vector store with guided index creation, KNN similarity search, hybrid search (dense + BM25 sparse), re-ranking with the cross-encoder ms-marco-MiniLM-L-6-v2 to improve top-K precision, prompt engineering patterns (system prompt template, context injection with citation tracking, chain-of-thought reasoning, output JSON schema enforcement), graceful degradation if Ollama is down (deterministic fallback to the raw top-K candidates + user-facing warning, no automatic cloud fallback for privacy consistency), P50/P95/P99 latency benchmarks with locust + markdown report, observability with end-to-end OpenTelemetry tracing, and a hardened systemd unit for deployment (NoNewPrivileges, ProtectSystem=strict, PrivateTmp). The philosophy is 'AI on-prem first': it shows that quality RAG can be done with Ollama + RediSearch without cloud vendors, reducing costs to €0/month and protecting sensitive data (GDPR, EU data residency). The chosen domain is a knowledge base of technical documents (articles, ADRs, manuals), close enough to the real case to be immediately useful, generic enough to be adapted to different enterprise contexts. MIT license, reusable production-grade code.

Key Features

  • Ollama on-prem integration

    Client HTTP for Ollama API, model warm-up, streaming response support, configurable fallback chain (llama3.1:8b → llama3.2:3b).

  • Sentence Transformers Embeddings Pipeline

    MiniLM-L6-v2 model, 384-dim, batch embedding async, caching embedding by content-hash, sentence-aware document chunking.

  • RediSearch HNSW Vector Store

    Index creation with HNSW parameter tuning (M, efConstruction, efRuntime), KNN search, metadata filtering, hybrid search BM25 + dense.

  • Ranking Model Re-Ranker

    ms-marco-MiniLM-L-6-v2 for re-ranking top-K candidates, controlled latency budget, A/B test re-rank on/off.

  • Gradual decline

    If Ollama goes down, endpoint returns deterministic result (raw top-K candidates) with a warning 'AI unavailable', no crash, and no automatic cloud fallback.

  • Locust benchmark suite

    Load test with Locust, latency (P50, P95, P99) reporting, throughput RPS, and memory footprint monitoring.

  • Prompt Engineering Patterns

    System prompt template with context injection, citation tracking, chain of thought reasoning, and enforced JSON schema output.

  • Deploy production-ready

    Systemd unit hardening (NoNewPrivileges, ProtectSystem=strict), environment variable externalization, log in structured JSON, health check endpoint with Olamma liveness.

Tech stack

  • FastAPI 0.115+
  • Python 3.12
  • Ollama
  • Redis Stack 7
  • RediSearch HNSW
  • sentence-transformers
  • cross-encoder
  • asyncio
  • Pydantic v2
  • locust
  • OpenTelemetry

Use cases

  • Developers who want to learn RAG without relying on OpenAI or Anthropic
  • Enterprise team with GDPR/data residency constraints seeking 100% on-premises AI stack
  • Tech enthusiasts with home labs seeking a private knowledge base powered by AI
  • PMIs in the EU evaluating AI on-prem as an alternative to SaaS (cost reduction and privacy)

Road Map

  1. v0.1.0Scheduled

    Baseline RAG: Embedding + Vector Search

    • Setup Olma and sentence-transformers
    • Rebuild Search Index
    • Endpoint /embed and /search
    • Chunking strategy - sentence aware
  2. v0.5.0Scheduled

    Rank by Type + Hybrid + Evolution

    • Cross-encoder re-ranking pipeline
    • Hybrid search (BM25 + dense)
    • Generation endpoint with prompt template
    • Graceful degradation on Yoda down
  3. v1.0.0Scheduled

    Production Hardening + Benchmark

    • Benchmark suite with Locust and Markdown Report
    • End-to-End Tracing with OpenTelemetry
    • Secure Deployment Guide for Systemd Units
    • Complete MkDocs Documentation

Who is it for

  • Junior/ Mid-Level Developer
  • For Developers - Senior

Quick start

Start the project locally in just a few commands. 

# Quick start (work-in-progress, repo non ancora pubblicato)
# git clone https://github.com/<tba>/fastapi-rag-ollama.git
# cd fastapi-rag-ollama
# ollama pull llama3.1:8b && ollama pull nomic-embed-text
# docker compose up -d redis-stack
# uv venv && uv pip install -e .[dev]
# uvicorn app.main:create_app --factory --reload

Contribute to the project

FastAPI RAG Ollama is open source. Open an issue, propose a pull request or leave a star on GitHub: every contribution, no matter how small, is welcome.

  • ⭐ Leave a star if it's useful
  • 🐛 Report bugs or suggest features via issue
  • 🔧 Send a pull request with fixes or improvements
  • 📣 Share the project if you find it interesting
Star on GitHubOpen an issuePull request
← ⭐ View on GitHub