Back up and restore

Data store

PostgreSQL (with the pgvector extension) is the single backing store, so a backup is one pg_dump. Both deployment profiles (lite, standard) share the same Postgres tables — long-term memories, short-term rows, auth/session state, and reindex job rows all live there.

Store	Technology	Persistence	Backup method
Memories	PostgreSQL + pgvector	Permanent (LTM + schema)	pg_dump --format=custom

Postgres is the source of truth. The pgvector index (the runtime-managed embedding_vec column) is derived and rebuildable with pnpm --filter mcp-server reindex, but a full pg_dump captures it along with everything else — one dump is the whole backup.

Quick start

# Take a full backup
DATABASE_URL=... ./scripts/backup.sh --out /var/backups/engram

# Apply retention policy (keep 30 days daily, 12 weeks weekly)
BACKUP_DIR=/var/backups/engram ./scripts/retention.sh

# Restore from an archive
DATABASE_URL=... \
  ./scripts/restore.sh --archive /var/backups/engram/engram_backup_20260601_020000.tar.gz

Retention policy

Configured via env vars or --flags to scripts/retention.sh:

Variable	Default	Meaning
BACKUP_RETENTION_DAYS	30	Daily backups retained for 30 days
BACKUP_RETENTION_WEEKLY	12	One backup per week for 12 weeks

Archives older than 12 weeks are deleted. Set BACKUP_RETENTION_DAYS=0 to skip the daily window and go straight to weekly rotation.

Automating with cron

# Daily backup at 02:00
0 2 * * * DATABASE_URL=... /opt/engram/scripts/backup.sh --out /var/backups/engram >> /var/log/engram-backup.log 2>&1

# Offsite sync at 02:30 (after the backup, before local pruning)
30 2 * * * rclone sync /var/backups/engram remote:engram-backups --include "engram_backup_*.tar.gz" >> /var/log/engram-offsite.log 2>&1

# Retention cleanup at 03:00
0 3 * * * BACKUP_DIR=/var/backups/engram /opt/engram/scripts/retention.sh >> /var/log/engram-retention.log 2>&1

Order matters: sync offsite before retention.sh prunes locally, so an archive is never deleted from the only copy that has it.

Offsite replication (3-2-1)

backup.sh writes to local disk only. A disk or host loss therefore takes the backups down with the data unless archives are replicated offsite. Aim for 3-2-1: three copies, two media, one offsite.

Sync the archive directory to object storage after every backup run, e.g.:

# rclone (any S3/GCS/B2/Azure remote)
rclone sync /var/backups/engram remote:engram-backups \
  --include "engram_backup_*.tar.gz"

# or the AWS CLI
aws s3 sync /var/backups/engram s3://<bucket>/engram-backups \
  --exclude '*' --include 'engram_backup_*.tar.gz'

Recommendations:

• Retention at the destination — apply lifecycle rules on the bucket (or run retention.sh against a synced mirror); rclone sync propagates local deletions, so a bucket with versioning/lifecycle is the safety net.
• Encrypt before upload when the bucket is not already encrypted with a customer-managed key: age -r <recipient> -o <archive>.age <archive> (or gpg --encrypt). Postgres dumps contain user memory content.
• Restrict credentials — the sync job only needs write/list on the backup prefix; use a scoped key, never the deployment’s admin credentials.
• Verify restorability, not just existence — the nightly backup verification workflow proves the scripts round-trip; periodically restore an offsite archive into a scratch environment to prove the offsite copies do too.

Continuous verification in CI

Two layers of automated coverage:

• Every PR / push — backup-restore.spec.ts (mcp-server suite) checks that backup.sh produces an archive and restore.sh round-trips the data. Beyond a legacy sentinel table, it now seeds and asserts the WP2-4 tables — memory_links, memory_audits, and memory_import_sources — so a dump that silently drops them (e.g. a future --table allowlist) reddens the PR (G9). retention.sh pruning is covered by backup-scripts.spec.ts against fake aged archives.
• Nightly — backup-verify.yml (cron + manual workflow_dispatch) runs the full backup.sh → restore.sh path against a throwaway Postgres service container: provisions the real ENGRAM schema, seeds the data (including the same WP2-4 tables), backs up, destroys the live data, restores, asserts every sentinel round-trips, and exercises the retention.sh GFS policy. A red run means the restore path is broken — treat it like a production incident, not a flaky test.

Restore procedure

./scripts/restore.sh \
  --archive /var/backups/engram/engram_backup_YYYYMMDD_HHMMSS.tar.gz

restore.sh reads DATABASE_URL and restores the Postgres dump from the archive. It prompts for confirmation before overwriting any data; pass --no-confirm in unattended environments.

To rebuild only the derived vector index (without a data restore), reindex from Postgres instead — see Disaster recovery below.

Verify a restore

After restoring Postgres, run the health check and validate memory counts:

curl http://localhost:3000/health
curl http://localhost:3000/health/ready

# Count memories via MCP tool (requires admin token)
curl -X POST http://localhost:3000/mcp \
  -H 'Content-Type: application/json' \
  -d '{"jsonrpc":"2.0","method":"tools/call","params":{"name":"list_memories","arguments":{"userId":"<user>"}},"id":1}'

Disaster recovery

Scenario: Postgres lost

1. Provision a new Postgres instance with pgvector/pgvector:pg17.
2. Restore the most recent archive with restore.sh (above). It drops and recreates the public schema, then pg_restores the custom-format dump — tables, indexes, and the embedding_vec column come back with the data, so no separate migration step is needed.

Without a backup this is a total loss event — Postgres is the source of truth. The most you can recover is an empty schema (pnpm db:migrate:deploy) to start fresh.

Recovery time objective (RTO): < 30 minutes. Recovery point objective (RPO): time since last successful backup.

Scenario: vector index lost or corrupted

The pgvector index (embedding_vec) is derived, so it never needs a restore — rebuild it from Postgres:

1. Stop ingest if needed.
2. Run pnpm --filter mcp-server reindex (or use the reindex_memories MCP tool).
3. The index is rebuilt from the stored embeddings — no data loss.

Schema for backup archives

Archives are named engram_backup_YYYYMMDD_HHMMSS.tar.gz and contain a single timestamped directory with the Postgres dump:

YYYYMMDD_HHMMSS/
  postgres.pgdump      pg_dump custom-format dump