Disaster recovery + backup strategy AI on-prem 2026: RPO RTO Hetzner

If the server is gone tomorrow morning: how much you lose and how long it takes to restart

Imagine a firm of forty people that has put an assistant on its own documents in-house: a database with users and cases, a cache, an index for semantic search and the model files. One morning the disk does not respond. Two numbers decide whether it is an incident or a crisis: how much work you have lost (RPO, Recovery Point Objective) and how much time passes before the assistant answers again (RTO, Recovery Time Objective). They are not technical numbers: they are chosen by whoever answers for the business, and the technique serves to respect them and to demonstrate that they can be respected.

There is also a regulatory reason to have them in writing. The GDPR, in Article 32 on security of processing, lists among the measures "the ability to restore the availability and access to personal data in a timely manner in the event of a physical or technical incident" and "a process for regularly testing, assessing and evaluating the effectiveness of technical and organisational measures for ensuring the security of the processing"; the NIS2 Directive, in Article 21(2), lists among the risk-management measures "business continuity, such as backup management and disaster recovery, and crisis management". NIS2 applies only to the entities that fall within its scope: whether your company does is to be established case by case, and this is not legal advice. The text that matters to everyone is the second quotation: a backup that has never been tested does not satisfy "a process for regularly testing".

This article lines up the choices for that constructed case: how to reason about RPO and RTO with a program, what to configure in PostgreSQL and Redis (with parameters verified on the official documentation), how to check with a program that was run that backups exist and are intact, how to encrypt them, and how to test the restore. The series has two premises: how much load the server sustains, in the article on Ollama from 1 to 50 users, and what has to be protected in terms of personal data, in Privacy by design. The traceability of who does what, which completes the picture, is in the next article on the audit log.

RPO and RTO: two numbers that change for every component

There is no single RPO for "the system". The assistant's database contains work that nobody can redo; a cache fills itself; a vector index is derived data, which is rebuilt from the database by recomputing the embeddings. The criterion for classifying each component is a single question: if I lose it, where do I regenerate it from and how much does it cost to do so? Where the answer is "nowhere" you need frequent backups, where it is "from another copy in a few hours" a nightly copy or none is enough.

The program below puts the reasoning into numbers for the constructed case. The first part lists the RPO in the worst case: with WAL archiving and an archive_timeout of 300 seconds you lose at most five minutes, without it you lose up to a nightly copy, that is a day. The second part adds up the recovery steps: those in sequence add up, those that can run in parallel count for the longest one. All the times are assumptions, to be replaced with the ones you measure in the restore drill further on.

"""RPO and RTO of an internal assistant with four components. All the times are ASSUMPTIONS to replace with the measurements from the restore drills."""

# RPO: how much work you can lose. It depends on the mechanism, not on how "frequent" the backup seems.
rpo_minuti = {
    "database, with WAL archiving": 5,          # archive_timeout = 300 s: incomplete segments are closed every 5 minutes
    "database, nightly copy only": 24 * 60,     # if you skip the WAL you lose up to a day
    "cache and sessions": 24 * 60,              # rebuildable: accepted
    "vector index": 24 * 60,                    # derived from the database: it is rebuilt
}
print("RPO in the worst case")
for nome, minuti in rpo_minuti.items():
    print(f"  {nome:34s} {minuti:5d} min")

# RTO: sequential recovery, the steps that can run in parallel count only once (the longest)
sequenza = [
    ("decide and declare the disaster", 10),
    ("provision or power on the machine", 15),
    ("restore the database base backup", 20),
    ("replay the WAL up to the chosen point", 10),
]
parallelo = [
    ("rebuild the vector index", 90),
    ("warm up the cache", 5),
]
tot_seq = sum(m for _, m in sequenza)
tot_par = max(m for _, m in parallelo)
print()
print(f"RTO for the database alone in service: {tot_seq} min")
print(f"RTO for full semantic search:          {tot_seq + tot_par} min")
obiettivo = 60
print(f"target of {obiettivo} min: database {'met' if tot_seq <= obiettivo else 'NOT met'}, "
      f"full search {'met' if tot_seq + tot_par <= obiettivo else 'NOT met'}")

# Output:
# RPO in the worst case
#   database, with WAL archiving           5 min
#   database, nightly copy only         1440 min
#   cache and sessions                  1440 min
#   vector index                        1440 min
#
# RTO for the database alone in service: 55 min
# RTO for full semantic search:          145 min
# target of 60 min: database met, full search NOT met

The result says something that the lone target "one hour" hides: the database comes back in 55 minutes and the target is met, but the assistant with full semantic search comes back in 145, because rebuilding the index (90 minutes in the assumption) is the longest step and depends on how many documents you have to reprocess. You have two roads: accept that semantic search comes back later than the rest, declaring it in the plan, or include the index in the backups so as not to have to rebuild it. Since I do not know how many documents you have, I do not know which one pays off: your measurement says.

The database: base backup, WAL and the proof that the restore works

The PostgreSQL documentation describes continuous archiving like this: a complete copy of the data directory (the base backup, obtained with pg_basebackup) plus the series of WAL segments generated afterwards. With the two things you can bring the database back to any instant after the base backup, not only to the moment of the copy. The parameter that makes the RPO meaningful is archive_timeout: archiving triggers only for complete segments, and if traffic is low the pause between a transaction and its copy to safety can be long; by setting the timeout, the server closes a segment at every interval, and sets a limit on the age of the unarchived data.

# Schema to adapt, NOT run: no PostgreSQL server was started for this article.
# postgresql.conf of the primary server (parameters from the official documentation, section "Continuous Archiving")
wal_level = replica
archive_mode = on
archive_command = 'test ! -f /mnt/archivio/wal/%f && cp %p /mnt/archivio/wal/%f'
archive_timeout = 300           # seconds: closes and archives a segment even if it is not full

The archiving command in the example is the one from the documentation. The line that matters is test ! -f ... && cp: the command must fail, that is return a non-zero code, if the file already exists, because PostgreSQL must never overwrite an archived segment and must retry until it succeeds. An archive_command that returns zero without having copied anything is the quietest way of losing a backup. To know whether archiving is working you need to look at the pg_stat_archiver view, which reports the last archived segment and the last failed one.

-- Schema to adapt, NOT run. To be launched every few minutes by the monitoring.
-- If last_failed_time is more recent than last_archived_time, WAL archiving is stopped.
SELECT last_archived_wal, last_archived_time, last_failed_wal, last_failed_time
FROM pg_stat_archiver;

The reading rule: if last_failed_time is more recent than last_archived_time, the archive is stopped and your real RPO is growing minute by minute, whatever the configuration says. It must be put under monitoring with an alarm, not looked at by hand. The restore, in turn, follows the instructions of the documentation section on recovery from a continuous archive: you extract the base backup, you indicate restore_command and a target point, you create the recovery.signal file and you start the server, which replays the WAL up to the chosen point. The schema below is to be adapted and was not run: it would have required starting a PostgreSQL server.

# Schema to adapt, NOT run. Restore from base backup + WAL (PostgreSQL documentation, "Recovering Using a Continuous Archive Backup").
# 1. Stop the server and empty the data directory; extract the base backup into the data directory
# 2. Tell the server where to find the archived WAL and until when to replay it (postgresql.conf or postgresql.auto.conf)
restore_command = 'cp /mnt/archivio/wal/%f %p'
recovery_target_time = '2026-11-12 14:55:00+01'
# 3. Create the file that starts the recovery and start the server
touch "$PGDATA/recovery.signal"
pg_ctl -D "$PGDATA" start

The target point (recovery_target_time) is the difference between a restore and a disaster within the disaster. If somebody deleted a table by mistake at 14:56, restoring to the last available instant brings back the deletion too; by choosing 14:55 you recover the state before the error. That is why a backup is not replaced by a replica: a replica copies the deletion in real time as well. Streaming replication serves a different goal, restarting quickly when the server stops, and is described in the section on standby servers of the same documentation: on the standby server you create the standby.signal file and indicate primary_conninfo.

# Schema to adapt, NOT run. Streaming replication (PostgreSQL documentation, "Log-Shipping Standby Servers").
# On the standby server: create the empty file standby.signal in the data directory, then in postgresql.conf
primary_conninfo = 'host=192.0.2.10 port=5432 user=replica application_name=riserva'
restore_command = 'cp /mnt/archivio/wal/%f %p'      # optional: it also reads from the archive if the connection is lost

# Check on the primary: the replication is running and how far behind it is
#   SELECT client_addr, state, replay_lsn FROM pg_stat_replication;
# Promotion of the standby to primary, when the primary is lost:
#   pg_ctl promote -D "$PGDATA"        (or SELECT pg_promote();)

The standby_mode parameter, which many guides still report, was removed in PostgreSQL 12 and replaced by the standby.signal file, as the release notes of that version say: if you find it in a guide, that guide is old. With the standby you can promote it to primary with pg_ctl promote or with pg_promote(), as the documentation says. Asynchronous replication has a delay: data written on the primary and not yet arrived at the standby are lost in the switch, so the RPO of a promotion is the replication lag at the moment of the failure, not zero. The pg_stat_replication view shows the state and position, and the lag must be monitored like that of the archive.

Redis: first decide what it is, cache or archive

If Redis in your system holds only caches and counters that are rebuilt, persistence is a comfort. If it also holds sessions, job queues or the search index, persistence is a requirement. The official documentation describes two mechanisms: RDB (snapshots at intervals) and AOF (a log of every write). It also says something useful for backups: the RDB file is never modified after it has been produced, and it is written with a temporary name and renamed atomically, so copying it while the server is running is safe. The documentation's suggestion is an hourly copy in one folder and a daily one in another, with find deleting the older ones.

# Schema to adapt, NOT run. redis.conf of the primary.
save 900 1                      # example: RDB snapshot if there is at least 1 change in 900 s
save 300 10                     # or 10 changes in 300 s
save 60 10000                   # or 10000 changes in 60 s
appendonly yes
appendfsync everysec            # fsync once a second: up to ~1 second of writes can be lost
# On the replica, to promote it to primary in case of failure (from redis-cli):
#   REPLICAOF NO ONE

The three save lines say: snapshot if there is at least one change in 900 seconds, or ten in 300, or ten thousand in sixty. appendfsync everysec makes the log be written to disk once a second: the documentation says it is fast enough and that in case of disaster a second of data can be lost. Since Redis 7 the AOF is no longer a single file but a set of files (a base file and incremental files) in a folder: if you copy an old appendonly.aof by hand you are copying the wrong thing. On the replica, to promote it, the command is REPLICAOF NO ONE, which the documentation describes as the way to become primary again; the old name SLAVEOF is deprecated.

I have not verified how an RDB backup treats the vector search index, and the text does not claim it: in the restore drill check with your own index-state command (FT.INFO returns information and statistics) that the number of documents is the expected one. If it is not, the road is to rebuild from the database. In no case use FLUSHDB as a quick remedy: it deletes all the keys of the selected database, not just the index.

Models and index: derived data, with a hidden dependency

The model files contain no state of yours and can be downloaded again. But "can be downloaded again" is a dependency on the model registry at the moment of the disaster, and on your bandwidth: the time is the file size divided by the line speed. For a 7.25-billion-parameter model at 4 bits, the article on sizing calculates a theoretical minimum of about 3.4 GiB: the download time is computed with your line. If the restore has to work even when the registry is unreachable, keep a copy of the model files you use in the backup archive, with their fingerprints, and note the exact version: the judge model of RAG evaluation and the generation model must come back identical, or the previous scores are no longer comparable.

The vector index is derived from the database, but rebuilding it costs time (in the program, 90 minutes of assumption) and depends on the embedding model: if the model changes, the embeddings are no longer compatible and everything must be rebuilt. That is why it is worth recording with every backup the name and version of the embedding model used. The configuration of the services (systemd units, Nginx, database parameters) is not data: it is code, and its backup is a versioned repository, not a script that copies folders.

An unverified backup is a hypothesis

The most common failure of backups is not the fire but silence: the script runs, writes a zero-byte file, and nobody notices for six months. Three automatic, inexpensive checks prevent it: every file has its fingerprint recorded in the manifest and it matches; the most recent backup is not older than the RPO; no file is much smaller than the median. The program runs the three checks on a temporary folder in which it builds seven example backups, then damages them in three typical ways: a truncated file, a file altered without changing its size, a missing file. The clock is a parameter, so the output is the same on every run.

"""Check of a backup folder: SHA-256 manifest, freshness against the RPO, anomalous sizes."""
import hashlib
import os
import tempfile
from datetime import datetime, timedelta
from statistics import median

ADESSO = datetime(2026, 11, 13, 6, 0, 0)       # passed as a parameter: no dependence on the clock


def sha256(percorso):
    h = hashlib.sha256()
    with open(percorso, "rb") as f:
        for blocco in iter(lambda: f.read(65536), b""):
            h.update(blocco)
    return h.hexdigest()


def controlla(cartella, manifesto, adesso, rpo, tolleranza=0.5):
    problemi = []
    dimensioni = {}
    for nome, atteso in manifesto.items():
        p = os.path.join(cartella, nome)
        if not os.path.exists(p):
            problemi.append(f"{nome}: MISSING")
            continue
        dimensioni[nome] = os.path.getsize(p)
        if sha256(p) != atteso:
            problemi.append(f"{nome}: fingerprint differs from the manifest")
    if dimensioni:
        mediana = median(dimensioni.values())
        for nome, d in dimensioni.items():
            if d < mediana * tolleranza:
                problemi.append(f"{nome}: {d} bytes against median {mediana:.0f}")
        piu_recente = max(datetime.fromtimestamp(os.path.getmtime(os.path.join(cartella, n))) for n in dimensioni)
        if adesso - piu_recente > rpo:
            problemi.append(f"last backup {adesso - piu_recente} ago: beyond the RPO of {rpo}")
    return problemi


with tempfile.TemporaryDirectory() as d:
    manifesto = {}
    for giorno in range(7):
        nome = f"db_{giorno}.sql.gz"
        contenuto = (f"line {giorno} " * 2000).encode()
        with open(os.path.join(d, nome), "wb") as f:
            f.write(contenuto)
        manifesto[nome] = hashlib.sha256(contenuto).hexdigest()
        quando = (ADESSO - timedelta(days=6 - giorno, hours=3)).timestamp()
        os.utime(os.path.join(d, nome), (quando, quando))

    print("1) intact folder:", controlla(d, manifesto, ADESSO, timedelta(hours=26)) or "no problem")

    # A truncated file (disk full halfway through writing) and one altered without changing its size
    with open(os.path.join(d, "db_6.sql.gz"), "wb") as f:
        f.write(b"x" * 300)
    t6 = (ADESSO - timedelta(hours=3)).timestamp()
    os.utime(os.path.join(d, "db_6.sql.gz"), (t6, t6))   # the rewritten file gets its time back: the example does not depend on the clock
    alterato = os.path.join(d, "db_2.sql.gz")
    dati = bytearray(open(alterato, "rb").read())
    dati[10] ^= 0xFF
    open(alterato, "wb").write(bytes(dati))
    os.remove(os.path.join(d, "db_4.sql.gz"))
    print("2) after the damage:")
    for p in controlla(d, manifesto, ADESSO, timedelta(hours=26)):
        print("  ", p)

    print("3) same state, three days later:")
    for p in controlla(d, manifesto, ADESSO + timedelta(days=3), timedelta(hours=26)):
        if "RPO" in p:
            print("  ", p)

# Output:
# 1) intact folder: no problem
# 2) after the damage:
#    db_2.sql.gz: fingerprint differs from the manifest
#    db_4.sql.gz: MISSING
#    db_6.sql.gz: fingerprint differs from the manifest
#    db_6.sql.gz: 300 bytes against median 14000
# 3) same state, three days later:
#    last backup 3 days, 3:00:00 ago: beyond the RPO of 1 day, 2:00:00

The first line confirms that the intact folder produces no false alarms. After the damage, the fingerprint check finds the altered file (db_2), which the size check would not have seen, and the truncated file (db_6), which is reported twice, for fingerprint and for size; the missing file appears as such. The third test moves the clock forward by three days and the freshness check fires. What the program cannot say is whether the content can be restored: the fingerprint proves that the file is the one that was written, not that it is a valid backup. Only the restore drill demonstrates that.

Encrypting backups: the key is not where the backups are

A backup contains the same personal data as the live system, and in addition it sits in a place that often has weaker protections. The same Article 32 of the GDPR cites the encryption of personal data among the appropriate measures. The rule that weighs more than the choice of tool is key management: if it is only on the server you have lost, the encrypted backups are a useless pile of bytes; if it sits with the backups, encryption protects nothing. The key must be kept elsewhere, and it must be tested as much as the backup.

The program that follows uses GnuPG with AES-256 symmetric encryption in a temporary folder with an isolated keyring, encrypts a file, decrypts it and compares the fingerprint with the original, then tries a wrong key. The passphrase is a test one and must not be reused: in production it lives in the secrets manager. The size of the encrypted file is not printed because it varies from one run to the next.

#!/bin/bash
# Symmetric encryption of a backup with GnuPG and a restore test. Isolated environment in a temporary folder.
set -euo pipefail
LAVORO=$(mktemp -d)
export GNUPGHOME="$LAVORO/gnupg"; mkdir -m 700 "$GNUPGHOME"
printf 'test-key-do-not-reuse' > "$LAVORO/passphrase"      # in production it lives outside the backup server

head -c 100000 /dev/zero | tr '\0' 'a' > "$LAVORO/db.sql"
sha256sum "$LAVORO/db.sql" | cut -d' ' -f1 > "$LAVORO/db.sha256"

gpg --batch --quiet --pinentry-mode loopback --passphrase-file "$LAVORO/passphrase" \
    --symmetric --cipher-algo AES256 --output "$LAVORO/db.sql.gpg" "$LAVORO/db.sql"
echo "encrypted: the plaintext does not appear in the file (occurrences of 'aaaa': $(grep -c aaaa "$LAVORO/db.sql.gpg" || true))"

gpg --batch --quiet --pinentry-mode loopback --passphrase-file "$LAVORO/passphrase" \
    --decrypt --output "$LAVORO/ripristinato.sql" "$LAVORO/db.sql.gpg"
if [ "$(sha256sum "$LAVORO/ripristinato.sql" | cut -d' ' -f1)" = "$(cat "$LAVORO/db.sha256")" ]; then
  echo "restore: fingerprint identical to the original"
fi

printf 'wrong-key' > "$LAVORO/sbagliata"
if gpg --batch --quiet --pinentry-mode loopback --passphrase-file "$LAVORO/sbagliata" \
    --decrypt --output /dev/null "$LAVORO/db.sql.gpg" 2>/dev/null; then echo "ERROR: wrong key accepted"; else echo "wrong key: decryption refused"; fi
rm -rf "$LAVORO"

# Output:
# encrypted: the plaintext does not appear in the file (occurrences of 'aaaa': 0)
# restore: fingerprint identical to the original
# wrong key: decryption refused

The output shows the three things that an encryption procedure needs: the plaintext does not appear in the encrypted file, the restore returns a file with an identical fingerprint, and the wrong key is rejected. It is an example tool, not a single recommendation: restic, which encrypts and deduplicates the data in the repository, uses AES-256 in CTR mode with a Poly1305-AES authentication code for integrity, as its documentation describes, and has the advantage of incremental backups. I did not run it here. Whatever tool you choose, test the restore with only the key you have kept elsewhere.

The restore drill: the only way to know your RTO

The GDPR asks for a process for regular testing, and your RTO, until you have measured it, is a wish. A restore drill is done on a test machine, with a stopwatch, following the written procedure and not the memory of whoever wrote it. The order of the steps matters, and the stopwatch marks the time of each: they are the numbers that replace the assumptions of the first program.

1. Power on a clean machine, with no access to production data except through the backup archive, and start counting.
2. Retrieve the key from the place where you keep it outside the server and decrypt the latest base backup.
3. Restore the database with base backup and WAL up to a chosen point and verify that the number of records in a known table is the expected one.
4. Restore Redis from the latest RDB copy and check the state of the index; if it is not intact, rebuild it from the database and note how long it takes.
5. Bring the models back from the backup archive or from the registry and verify the fingerprints.
6. Run a dozen questions from the golden dataset on the restored system and compare the scores with the latest measurement.
7. Write down the time of every step and what went wrong, then update the plan and the programs' assumptions.

The last step is the one that almost everybody skips. The list is in order because each step depends on the previous one: without the key you cannot decrypt, without the database there is nothing to query. The golden dataset questions, born in the article on evaluation, serve here as a functional test: a system that starts but answers differently from before is not restored. If the drill takes more than the declared RTO, you do not have a script problem but a plan problem, and the correction may be to change the target, not only the system.

Frequently asked questions

Does a replica replace the backup?

No. The replica copies an error too, such as a deleted table, at the moment it happens. It serves to restart quickly if the server stops, not to go back in time, which is what the base backup and the WAL archive do.

What RPO do I choose for an assistant on documents?

It depends on how much work you can afford to redo. For the database, with WAL archiving and an archive_timeout of 300 seconds, the worst case is about five minutes. For cache and index, which are rebuilt, a daily copy or none is usually accepted.

How often should I test the restore?

The regulation asks for regular verification and does not fix the cadence. A drill at every important change of the infrastructure and at least once a quarter is a reasonable choice, but it is your choice, to be written in the plan.

Can I copy the RDB file while Redis is working?

Yes: the official documentation explains that the RDB file is not modified after production and that it is renamed atomically when the snapshot is complete, so copying with the server on is safe.

What you can do now

You can say how much work you lose and how long it takes to restart, component by component, and you can drop the targets when a measurement contradicts them. You can configure PostgreSQL WAL archiving with a limit on the age of unarchived data, you know that a replica does not replace a backup and that the restore point is chosen. You can check that backups exist, are fresh and intact, encrypt them without losing the key and test the restore with a stopwatch. The next step is to demonstrate, with a log, what the system has done: audit log and traceability.

The same series also includes the cost calculation for a server of your own, the cases where a managed service is the better choice and the recap with the decision sheet.