Off-site backups (Admin > System health)

Signed in? Ask Lovelio this question inside the app - it answers from this same page.

Every night, in its own region's small hours, each database (US, EU, ANZ and the platform database behind lovelio.ai) is exported with pg_dump, encrypted and written to S3, where it is locked against deletion for 35 days. Once a month each export is restored into a fresh Postgres and the tables and row counts are compared with the manifest, which proves the file can actually be put back. Small databases restore into Docker on the GitHub runner for free; a database too big for the runner's disk (ANZ, 43 GB) restores into a fresh Supabase project in its own region, sized from the export's measured database size, and that project is deleted when the run ends, so the drill costs cents in compute, never a monthly project.

The hourly check emails an alert when a backup breaks, when a new problem appears and when everything recovers. While the same problem stays broken it repeats once a day, not every hour.

The Off-site backups section of System health (/admin/health) shows one line per database, from the same assessment the hourly cron alerts on:

Export: Healthy means last night's export landed. Overdue means the last good export is older than the schedule allows. Failing means the last attempt errored, and the error is printed under the badge; a Failing export with no last good time has never succeeded, and the error says why. Never run means the export workflow has never reported in for that database, which is a workflow that is disabled or cannot start.
Last good export: when, and the date.
Size and Tables / rows: what the last good export contained.
Restore test: the same four verdicts for the monthly restore into a fresh Postgres, with the last error printed when it failed.

Press Refresh to re-read. Nothing on this page changes anything; it is a health readout.

What to do when a line is not green

Export Overdue or Failing: read the error. The usual causes are the Supabase project being paused, the database password rotated, the export running out of disk, or GitHub refusing to start jobs because its bill failed or the Actions spending limit was reached (the run says so under Annotations). The export looks again every hour through the night, five times, and goes ahead at the first quiet hour or the fifth regardless, so a transient failure retries within the hour and a missed night retries the next.
Restore test Failing or Never run: the error names the table or the resource that ran out. A restore test that has never passed means the backup is unproven, which is the alert to act on first. The GitHub runner has one disk: the workflow clears the toolchains it never uses and refuses to start under 35 GB free, and the script checks the dump fits before it downloads, so "No space left on device" now points at the runner, not the backup. A database the runner cannot hold goes to a drill project instead, chosen per run from the measured sizes; a drill project left behind by a run that died is deleted by the next run's sweep.

How this differs from Admin > Restore

Restore (/admin/restore) puts ONE agency back to a minute using the 30-day change history in the live database. Off-site backups are the last line: a copy of the whole database outside Supabase, for the day the live database itself is gone. Bringing one back is a runbook, not a button.