Skip to content

Backups and restore

What is backed up

tempest-prod's scheduler runs python manage.py daily_maintenance at 03:30 Europe/London, and shortly after start if the last run is more than 25 hours old. Each run:

  • writes a backup folder under data/prod/backups/ named by its UTC time, for example 2026-10-21T0230Z/:
  • db.dump: pg_dump in custom format (PostgreSQL 16 client, matching the server)
  • media/: the stored 2048-pixel JPEGs and 480-pixel thumbnails
  • manifest.json: record counts and sizes, written last
  • deletes backup folders older than 14 days
  • permanently removes Trash older than 30 days and leaves deletion markers for devices
  • removes deletion markers every recently seen device has synced past, or that are older than 30 days
  • purges accounts whose deletion date has passed, and lifts suspensions that have expired
  • prunes the forecast cache (older than a day), idempotency records (60 days) and edit history (90 days)

A folder is only renamed from ….partial to its final name once everything in it is written, so a folder without .partial is a complete backup. Stored photos never change once written, so a file that is unchanged since the previous backup is a hard link to it. Every backup folder is still a complete tree you can copy on its own.

The last run's summary is in data/prod/backups/.last_maintenance.json. To run it by hand:

docker compose -f compose.prod.yaml exec web python manage.py daily_maintenance

Copying data/prod/backups off this machine is your own process and is what protects against losing the machine. Docker volumes and this folder alone do not.

Restoring

Run from /home/ajoyner/tempest:

scripts/restore_backup.sh prod data/prod/backups/2026-10-21T0230Z

The script:

  1. stops web and scheduler, and makes sure db is running
  2. drops and recreates the database, then loads db.dump with pg_restore
  3. replaces data/<env>/media with the backup's media/
  4. runs python manage.py after_restore, which moves the sync sequence far ahead and makes every phone and browser download a fresh catalogue on its next sync (their cursors may be newer than the restored database)
  5. starts the stack again with up -d

The same script restores into dev (scripts/restore_backup.sh dev …), which is how a production backup can be checked without touching production.

Rebuilding on a new machine

  1. Clone the repository to /home/ajoyner/tempest, copy in .env.prod, and copy the backup folder into data/prod/backups/.
  2. mkdir -p data/prod/postgres data/prod/media and docker compose -f compose.prod.yaml build.
  3. docker compose -f compose.prod.yaml up -d db, wait for PostgreSQL to accept connections, then run the restore script.

What a restore loses

A restore returns the server to the moment of the backup. Anything that reached the server after that moment is not in the backup. Phones keep captures that never uploaded and send them after the fresh download. Records that had already synced after the backup are dropped from phones when they re-download, because the server no longer has them.