Skip to main content

Backup & Restore

Back up a production Hrida.ai Studio deployment: continuous PostgreSQL backups with WAL-G, encrypted off-server copies of files, volumes and LDAP with the hrida-backup sidecar, and how to restore each one.

The production stacks (docker-compose-studio-prod.yaml, and the Docker Swarm stacks) back up in two layers. Each runs from a container image of its own:

LayerImageWhat it coversWhere the copy goes
Database backupshrida-ai-studio-postgresThe main PostgreSQL database: users, chats, settings, Keycloak, and embeddings with pgvectorThis server's MinIO by default, or any S3 bucket
Off-server backupshrida-ai-studio-backupEverything else: the files in MinIO (including the database backups), the app's data volume, LDAP, GrafanaAn S3 bucket on another server, encrypted

Both are off until you turn them on. Turn on both: the database backup alone stays on the same server, so it doesn't survive losing the server.

Single container

If you run Studio as one container (docker run) with its data in one volume, back up that volume as shown under "Backup & Restore" in Updating.

Database backups (WAL-G)​

The postgres service runs hrida-ai-studio-postgres: the stock pgvector/pgvector:pg16 image plus WAL-G. Until you turn backups on, it behaves exactly like the stock image.

Disaster recovery, not high availability

The backup protects against a lost or corrupted database volume. There's no failover or standby: while Postgres is down, Studio is down.

What it covers​

DataStored inBacked up by WAL-G?
Users, chats, settings, artifacts and Keycloak's dataPostgreSQLYes
Embeddings, with VECTOR_DB=pgvectorThe same databaseYes
Embeddings, with Chroma (the default VECTOR_DB)The hrida-ai-studio volumeNo: hrida-backup covers it
Uploaded filesMinIO, or the hrida-ai-studio volumeNo: hrida-backup covers it

By default the backups go to the stack's own MinIO, in the hrida-pg-backups bucket. That covers a lost database volume, but not a lost server or disk. For that, turn on hrida-backup, which copies the bucket to another server every hour, or point WAL-G at an off-server bucket (settings).

How it works​

With STUDIO_PG_BACKUP=true (only the exact value true; anything else keeps the stock behaviour):

  • Continuous archiving. Every change is shipped to the bucket within about 60 seconds, so at most about a minute of changes can be lost.
  • Nightly full backups at 03:00, keeping the last 7. If there's no backup yet, the first runs as soon as Postgres is up, so the database is protected from the moment you turn this on, not from the next night.
  • Automatic restore. When the container starts on an empty data volume, it fetches the latest full backup, replays the changes archived since, and continues as a normal read-write database.
  • No silent fresh start. On an empty volume, if the bucket can't be reached within 120 seconds, the container refuses to start. It never creates an empty database where a backup could have been restored.

The one-off minio-init service creates the bucket, and Postgres creates it too if it's missing.

Settings​

In .env:

VariableDefaultWhat it does
STUDIO_PG_BACKUP(empty)true turns backups on
WALG_BACKUP_CRON0 3 * * *When full backups run (cron syntax)
WALG_RETAIN_FULL_BACKUPS7How many full backups to keep
WALG_S3_PREFIXs3://hrida-pg-backups/postgresWhere the backups go
WALG_S3_ENDPOINThttp://minio:9000The S3 server
WALG_S3_ACCESS_KEY, WALG_S3_SECRET_KEYMinIO's root userThe S3 server's credentials
WALG_S3_REGIONus-east-1The bucket's region
STUDIO_POSTGRES_TAGlatestThe image tag

On Docker Swarm, the stack's Postgres always writes its backups to the stack's MinIO; hrida-backup copies them off the server.

Turn it on​

bash
# In .env: STUDIO_PG_BACKUP=true (and MINIO_ROOT_PASSWORD, if it isn't set yet)
docker compose -f docker-compose-studio-prod.yaml pull postgres minio minio-init
docker compose -f docker-compose-studio-prod.yaml up -d postgres minio minio-init
docker logs -f hrida-postgres      # wait for: backup-cron ... done

The image is PostgreSQL 16, like pgvector/pgvector:pg16, so the existing postgres-data volume is reused as it is.

On Docker Swarm, set STUDIO_PG_BACKUP=true in .env, load it into the shell (set -a; . ./.env; set +a) and redeploy the stack as usual.

Check it's working​

bash
docker exec hrida-postgres wal-g backup-list     # at least one base_... line
docker exec hrida-postgres wal-g wal-show        # changes archived, with no gaps
docker logs hrida-postgres 2>&1 | grep -i "archive command failed"   # should print nothing

On Docker Swarm, run these on the data node, with the container's name from docker ps.

Restore after data loss​

  1. Make sure the backups are intact: MinIO's data, or your off-server bucket. If the whole server is gone, restore MinIO first, as described in Whole server lost.

  2. Stop the app and Postgres, remove the broken data volume and start Postgres on an empty one:

    bash
    docker compose -f docker-compose-studio-prod.yaml stop hrida-ai-studio hrida-ai-studio-worker postgres
docker compose -f docker-compose-studio-prod.yaml rm -f postgres
docker volume ls | grep postgres-data          # the volume's exact name
docker volume rm <project>_postgres-data
docker compose -f docker-compose-studio-prod.yaml up -d postgres
docker logs -f hrida-postgres                  # "restoring latest base backup", recovery, then ready

  3. Start the app again: docker compose -f docker-compose-studio-prod.yaml up -d.

Rehearse it​

A backup you have never restored is only a hope. Rehearse on throwaway containers, never on the real volumes:

  1. Run the image with backups on against a scratch MinIO, and add some rows.
  2. Force a full backup (wal-g backup-push), then add more rows.
  3. Delete the data volume and start the container again.
  4. Check that the rows from both before and after the full backup are back.

Repeat it after updating the image.

Off-server backups (hrida-backup)​

The database backup covers only the main PostgreSQL database, and by default keeps it on the same server. The hrida-backup service runs hrida-ai-studio-backup and copies everything else to an S3 bucket on another server or provider, so your data survives losing the whole server.

What it covers​

DataHowEncryptedWhen
MinIO buckets: hrida-studio-files, hrida-pg-backups (the database backups) and any othersCopied object by object through MinIO's S3 API, never from the raw volumeNo (see below)Hourly
The hrida-ai-studio volume (/app/backend/data), except cache/SnapshotYes02:00
The LDAP directoryLDIF export over the networkYes02:00
The LDAP configuration (/etc/ldap/slapd.d)Snapshot, plus a cn=config export if LDAP_CONFIG_PASSWORD is setYes02:00
Grafana: grafana-data and lgtm-data (with docker-compose.otel.yaml)SnapshotsYes02:00
postgres-secondary, if it runs and holds datapg_dumpallYes02:00
KeycloakStored in the main database, so the database backup covers it——

Not backed up: redis-data (Valkey's cache and queues), ollama (models can be pulled again), and prometheus-data, loki-data and tempo-data (telemetry).

The copied MinIO objects are stored as they are, so any S3 tool can restore them: protect the destination bucket with its own access policy. Everything else goes into a restic repository in the bucket's restic/ folder, compressed and encrypted with BACKUP_RESTIC_PASSWORD. It keeps 7 daily, 4 weekly and 6 monthly snapshots.

How it behaves​

  • Off by default. Without BACKUP_ENABLED=true the container logs disabled and idles. If a setting is missing, the bucket doesn't exist or the restic password is wrong, it logs exactly what's wrong and idles as unhealthy. It never restarts in a loop, and no other service depends on it.
  • Refuses a destination on the same server. If BACKUP_S3_ENDPOINT is this server's MinIO, it doesn't back up: a name like minio or hrida-minio, a loopback address, the same host as BACKUP_MINIO_ENDPOINT, or a name that resolves to the same address.
  • The first backup runs at start, if there's none yet. After that, jobs follow their schedules, and a job never overlaps its own previous run.
  • Read-only. The volumes it backs up are mounted read-only. Its logs go to docker logs hrida-backup.
  • Health. The container turns unhealthy when a job's last success is older than 26 hours.
  • Deleted files stay in the copy; changed files replace it. A file deleted from MinIO stays in the copy unless BACKUP_MINIO_REMOVE=true. A file changed in place, for example encrypted by ransomware, overwrites its copy at the next hourly run. Turn on versioning on the destination bucket so earlier versions stay recoverable.

Because deletions aren't copied, the copy grows over time: it also keeps the old database backups that WAL-G removes. With versioning on the destination, you can set BACKUP_MINIO_REMOVE=true to keep the copy in step with MinIO; deleted and overwritten files then remain as earlier versions, which a lifecycle rule for noncurrent versions can expire.

Settings​

In .env:

VariableDefaultWhat it does
BACKUP_ENABLED(empty)true turns backups on
BACKUP_S3_ENDPOINT—The off-server S3 endpoint, for example https://s3.eu-central-1.amazonaws.com
BACKUP_S3_BUCKET—The destination bucket; create it first
BACKUP_S3_ACCESS_KEY, BACKUP_S3_SECRET_KEY—A key that can read and write the bucket
BACKUP_S3_REGIONus-east-1The bucket's region
BACKUP_RESTIC_PASSWORD—Encrypts the snapshots. Generate it with openssl rand -base64 32
BACKUP_MINIO_CRON0 * * * *When MinIO is copied (cron syntax); off turns the copy off
BACKUP_VOLUMES_CRON0 2 * * *When the volumes, LDAP and postgres-secondary are backed up
BACKUP_KEEP_DAILY, BACKUP_KEEP_WEEKLY, BACKUP_KEEP_MONTHLY7, 4, 6How many snapshots to keep
BACKUP_MAX_AGE_HOURS26The container turns unhealthy after this long without a successful run
BACKUP_EXCLUDESstudio-data/cachePaths left out of the snapshots, comma-separated
BACKUP_MINIO_ENDPOINThttp://minio:9000This server's MinIO, the source
BACKUP_MINIO_BUCKETS(all)Copy only these buckets, comma-separated
BACKUP_MINIO_REMOVEfalsetrue also removes the copies of files deleted from MinIO
BACKUP_PG_SECONDARY_HOST, BACKUP_PG_SECONDARY_USER, BACKUP_PG_SECONDARY_PASSWORDpostgres-secondary, postgres, —The secondary database
BACKUP_LDAP_URIldap://ldap:389The LDAP server; empty skips LDAP. The base DN comes from LDAP_DOMAIN
STUDIO_BACKUP_TAGlatestThe image tag
BACKUP_MEM_LIMIT512mThe container's memory limit

Turn it on​

  1. Create a bucket on an S3 service on another server or provider (another MinIO, AWS S3, Backblaze B2, Wasabi…) and an access key that can read and write it.

  2. In .env, set BACKUP_ENABLED=true, the BACKUP_S3_* values and BACKUP_RESTIC_PASSWORD.

    Keep a copy of the password somewhere else

    Store BACKUP_RESTIC_PASSWORD and the BACKUP_S3_* values outside this server, in a password manager. Without the password nobody can read the encrypted backups, including you.

  3. Start it:

    bash
    docker compose -f docker-compose-studio-prod.yaml pull hrida-backup
docker compose -f docker-compose-studio-prod.yaml up -d hrida-backup
# With Grafana, include its overlay so its volumes are backed up too:
docker compose -f docker-compose-studio-prod.yaml -f docker-compose.otel.yaml up -d hrida-backup
docker logs -f hrida-backup

On Docker Swarm​

docker-compose-common-app-prodSwarm.yaml runs hrida-backup on the node labelled hrida.data. Set the values in .env, load it into the shell (set -a; . ./.env; set +a) and redeploy the stack as usual.

  • It backs up the data node's copy of the hrida-ai-studio volume; each node has its own.
  • It reads the Grafana and ldap-config volumes by their ${VOLUME_PREFIX}_… names.
  • It exports LDAP from BACKUP_LDAP_URI, by default ldap://openldap:389; empty skips LDAP.
  • A volume that doesn't exist on that node shows as empty — skipped in the log.

Check it's working​

bash
docker logs hrida-backup 2>&1 | tail -n 40       # "done in ...s — saved: ..." for both jobs
docker inspect -f '{{.State.Health.Status}}' hrida-backup   # healthy
docker exec hrida-backup healthcheck.sh          # the age of each job's last success
docker exec hrida-backup restore.sh list         # the snapshots of each item, and the copied buckets
docker exec hrida-backup restore.sh check        # integrity check; reads 5% of the data
docker exec hrida-backup backup-volumes.sh       # back up the volumes now (backup-minio.sh: MinIO)

unhealthy with a config error: line from healthcheck.sh means a setting needs fixing. STALE means scheduled runs are failing, and the logs say why.

Restore​

Every restore runs in a one-off hrida-backup container that mounts the target read-write. restore.sh reads the same BACKUP_* values from .env, and works even while BACKUP_ENABLED is off.

The volume names in the -v options below are the names in the compose file. Compose maps them to the project's real volumes (for example hrida-ai-studio_hrida-ai-studio), so run the commands from the folder you deploy from, with the same -f files.

bash
C="docker compose -f docker-compose-studio-prod.yaml"    # add -f docker-compose.otel.yaml for Grafana
$C run --rm --no-deps hrida-backup restore.sh list       # pick a snapshot id, or use the latest
# The running sidecar mounts the volumes, and Docker won't remove a volume
# that's in use, so take it down for the restore:
$C rm -sf hrida-backup

When you're done, bring it back with $C up -d hrida-backup.

Whole server lost​

Restore in this order:

  1. Set up the new server with the same deployment files and .env, including all the BACKUP_* values, and create the network: docker network create hridaai-network.
  2. MinIO first. Run $C up -d minio minio-init, then restore the MinIO buckets. This also brings back the database backups in hrida-pg-backups.
  3. Then Postgres. With STUDIO_PG_BACKUP=true, $C up -d postgres starts on an empty volume and restores itself from the database backups you just brought back (Restore after data loss).
  4. Restore the volumes and LDAP, then start everything with $C up -d.

MinIO buckets​

bash
$C up -d minio
$C run --rm --no-deps hrida-backup restore.sh minio --all                 # every copied bucket
$C run --rm --no-deps hrida-backup restore.sh minio hrida-studio-files    # or just one

This copies <BACKUP_S3_BUCKET>/minio/<bucket> back into the MinIO at BACKUP_MINIO_ENDPOINT, creating the bucket if needed. Objects with the same name are overwritten, and nothing is deleted. To restore into another MinIO, pass RESTORE_MINIO_ENDPOINT, RESTORE_MINIO_ACCESS_KEY and RESTORE_MINIO_SECRET_KEY with -e.

App data and Grafana volumes​

Stop the services that use the volume, empty it (or start from a new volume), restore, and start the services again:

bash
$C stop hrida-ai-studio hrida-ai-studio-worker
$C rm -f hrida-ai-studio hrida-ai-studio-worker
docker volume rm <project>_hrida-ai-studio      # exact name: docker volume ls | grep hrida-ai-studio
$C run --rm --no-deps -v hrida-ai-studio:/restore hrida-backup restore.sh volume studio-data /restore
$C up -d hrida-ai-studio hrida-ai-studio-worker

Grafana works the same way: add -f docker-compose.otel.yaml to $C, stop grafana, and restore with -v grafana-data:/restore … volume grafana-data /restore and -v lgtm-data:/restore … volume lgtm-data /restore.

  • To restore an older snapshot, add its id after the target.
  • --force restores into a volume that isn't empty, overwriting files in place.
  • File owners and permissions are restored as they were.
  • cache/ isn't backed up; Studio downloads its models again when it needs them.

LDAP​

The LDIF export holds the directory's entries; the ldap-config snapshot holds slapd's configuration and schema. LDAP must be stopped, because slapadd writes the database files directly.

bash
$C stop ldap phpldapadmin && $C rm -f ldap
docker volume rm <project>_ldap-data <project>_ldap-config
# 1. The config volume, from its snapshot
$C run --rm --no-deps -v ldap-config:/restore hrida-backup restore.sh volume ldap-config /restore
# 2. The directory as LDIF, parents first, into a folder on the host
mkdir -p ./restore
$C run --rm --no-deps -v "$PWD/restore:/out" hrida-backup restore.sh ldap-ldif /out/ldap-data.ldif
# 3. Load it into the empty ldap-data volume, and give the files to slapd's user
$C run --rm --no-deps -v "$PWD/restore:/in:ro" --entrypoint sh ldap -c \
  'slapadd -F /etc/ldap/slapd.d -b "dc=hrida,dc=local" -l /in/ldap-data.ldif && chown -R openldap:openldap /var/lib/ldap'
$C up -d ldap phpldapadmin

Replace dc=hrida,dc=local with your LDAP_DOMAIN: example.com is dc=example,dc=com.

If there's no ldap-config snapshot (for example on Docker Swarm, where LDAP's volumes aren't on the data node), do this instead of step 1:

  1. Start ldap once on empty volumes, with the same LDAP_DOMAIN and LDAP_ADMIN_PASSWORD, so it creates a fresh configuration.
  2. Stop it and empty only the ldap-data volume.
  3. Continue from step 2 above.

restore.sh dump ldap-config-ldif <file> writes the cn=config export, to compare or re-apply custom configuration.

postgres-secondary​

bash
# Replay the dump into a running, empty server:
$C run --rm --no-deps -e RESTORE_PG_HOST=postgres-secondary \
  -e RESTORE_PG_USER=postgres -e RESTORE_PG_PASSWORD=... hrida-backup restore.sh pg-secondary
# Or only write the SQL file:
$C run --rm --no-deps -v "$PWD/restore:/out" hrida-backup restore.sh dump pg-secondary /out/pg-secondary.sql

The dump comes from pg_dumpall, so it also creates the roles and databases. A role "postgres" already exists error is expected and harmless.

Rehearse it​

docker/backup/rehearsal/rehearse.sh, in Studio's repository, runs the whole round trip on throwaway containers and volumes only. They live in the compose project hrida-backup-rehearsal and are deleted at the end. The script:

  1. Seeds a source MinIO, LDAP, a secondary database and test volumes.
  2. Checks that the sidecar stays idle when it's off; refuses missing settings, a destination on the same server (by name and by address) and a wrong restic password; and never restarts in a loop.
  3. Lets the first backup run on its own, then checks that:
    • every snapshot exists;
    • the destination holds no plain text;
    • an overlapping run is skipped;
    • the health check fails when the last success is older than 26 hours;
    • a stopped secondary database is skipped cleanly.
  4. Deletes all the source data, restores everything with restore.sh, and compares fingerprints: object hashes, file hashes with owners and permissions, row checksums and LDAP entries.
bash
docker login ghcr.io       # the MinIO image comes from GHCR (or set REHEARSAL_MINIO_IMAGE)
bash docker/backup/rehearsal/rehearse.sh     # KEEP=1 leaves the stack up afterwards

Run it again after updating the backup image.

Notes​

  • postgres-secondary is not a replica. In docker-compose-studio-prod.yaml it's a separate database that doesn't copy the main one, so it's no standby. It has no POSTGRES_PASSWORD, so it can't start on an empty volume. hrida-backup dumps it only when it accepts connections and holds data; otherwise the log says pg-secondary: ... skipped. If it was set up by hand with a password, set BACKUP_PG_SECONDARY_USER and BACKUP_PG_SECONDARY_PASSWORD. To replay a dump into it, give it a password first, or restore into any other PostgreSQL 16 server.
  • Keycloak keeps its data in the main database (KC_DB_URL_HOST=postgres), so the database backup covers it. On Docker Swarm, Keycloak is defined in docker-compose-common-enterprise-Studio.yaml: check that its KC_DB_URL_HOST points at the same Postgres.
Hrida.ai is proprietary software of Zlabs Innovation. See the license for terms. © 2026 Zlabs Innovation.