Back up and restore

Back up, verify, and restore the hub store as one unit

By the end you will have a verified backup of the whole data directory and a tested way to put it back. The store's shape is in Data model; this page is the procedure.

Before you start

Back up the whole directory as one unit. A copy of hub.db alone silently omits recent writes, because hub.db-wal is part of the data.

What a backup holds

Both backup kinds write the same set:

• every engine file, copied through the engine itself (VACUUM INTO);
• every artifact blob, verbatim;
• a manifest.json holding the schema version, a timestamp, and each file's size and SHA-256.

A backup is the whole set of files the manifest names. When the embedded tailnet is in use, its key state, tailnet/keys.json, is part of the set, so a restore keeps the node's tailnet identity instead of re-registering. A project's files that a delete moved aside and could not remove are left out: they are no longer part of the store, and the next start removes them. check and restore read either kind the same way.

Offline backup

agent-hub backup --out /path/to/backup

A running hub holds the engine's exclusive file lock, so the offline backup refuses while one serves the store, and says to ask the hub with --url, stop it, or snapshot the volume. With the store free it copies everything while it holds the lock itself, so nothing changes under the copy.

Online backup

The hub is the single writer for every file it serves, so it can take the backup itself without stopping. Online backup is off until backup_dir (HUB_BACKUP_DIR) names a directory for it, outside the data directory:

HUB_BACKUP_DIR=/srv/agent-hub-backups

The hub refuses to start with a backup_dir inside the data directory, and checks again with symlinks resolved before each backup. The directory must already exist: an unmounted backup volume is reported, not filled in on the root filesystem. Then ask the running hub for a backup:

HUB_ADMIN_TOKEN=<the admin token> agent-hub backup --url http://127.0.0.1:8080

The request is the admin's, so the token is HUB_ADMIN_TOKEN, or admin_token under [hub] in config.toml, the one the hub serves with. The hub writes a new directory named for the UTC time to the millisecond, such as 20261007T031500.123Z, under backup_dir, and nowhere else: the request carries no path. The command prints where the backup landed, and exits 0 on success, 1 when the hub refused (online backup off, one already running), 69 when the hub is unreachable, and 77 when the token was refused. HUB_TIMEOUT bounds the wait; a backup the client stops waiting for still runs to the end on the hub.

The same request without the binary, for a host that only has curl:

curl -fsS -X POST -H "Authorization: Bearer $HUB_ADMIN_TOKEN" \
  http://127.0.0.1:8080/api/v1/backups

It answers 201 with the directory's name, its path on the node, the manifest's schema version, timestamp, file count and bytes, and how long the backup took. Online backup off is a 404 saying how to turn it on, a backup already running is a 409, and a missing backup directory is a 503.

What an online backup guarantees

• Every engine file is a consistent snapshot of itself. hub.db is copied through the hub's own handle, inside one read transaction, so writers carry on and the copy holds exactly what was committed when it began. Each session brain and knowledge file is copied the same way while its per-file write lock is held, so no write to that file lands mid-copy.
• The store never names a blob the backup lacks. hub.db is copied first, and a blob is always written before the row that names it commits, so every blob the snapshot names is on disk by then. Deleting or overwriting those files is held off for the whole backup (an artifact or project delete, a prune's commit, a live artifact write and the update that seals it, which wait and then finish), so none of those blobs can go or change before it is copied, and each one holds the bytes its row describes. The hub reads the copied store back and refuses to finish a backup that names a blob it lacks.
• The set is not one instant across files. The brain, knowledge, and blob files are copied after hub.db, so they can hold writes the store snapshot does not: a page written to a brain moments after the snapshot, a blob whose row committed later, a session brain with no session row yet. The first start on a restored directory removes blobs and brain files no row names, as it does after a crash.
• One backup at a time, and nothing is deleted or written live while one runs.

Schedule it

There is no scheduler inside the hub. Run the request from cron or a systemd timer on the node, for example nightly at 03:15:

# crontab -e
15 3 * * * HUB_ADMIN_TOKEN=<the admin token> /usr/local/bin/agent-hub backup --url http://127.0.0.1:8080

or as a timer and a oneshot service:

# /etc/systemd/system/agent-hub-backup.service
[Service]
Type=oneshot
EnvironmentFile=/etc/agent-hub/backup.env
ExecStart=/usr/local/bin/agent-hub backup --url http://127.0.0.1:8080

# /etc/systemd/system/agent-hub-backup.timer
[Timer]
OnCalendar=*-*-* 03:15:00
Persistent=true

[Install]
WantedBy=timers.target

with HUB_ADMIN_TOKEN=... in /etc/agent-hub/backup.env, readable by the service's account only. Old backups are never deleted by the hub. You prune them, as you do sessions: remove the oldest directories under backup_dir when the volume needs the room.

Snapshots

On a NAS or a virtual machine the other alternative is a filesystem snapshot of the data directory, which is consistent without stopping the hub. Either way, remember that hub.db-wal is part of the data.

Verify

agent-hub check --data-dir /path/to/data

It verifies every file the manifest names (present, right size, right digest), runs PRAGMA integrity_check on the store and every brain and knowledge file, and cross-checks artifact rows against the blobs on disk. It exits non-zero on any problem. A store older than this binary's artifact history is checked without the cross-check, because there are no version rows to read.

Run this against a backup before you rely on it, and run it on the live data directory on a schedule. A partially corrupt store is found here, not by /readyz: the readiness probe runs the cheap legs only, and a zeroed page in a table the probe does not touch leaves it answering 200 while the routes that read that page fail with 503.

Restore

agent-hub restore --from /path/to/backup --data-dir /path/to/data

An offline and an online backup restore the same way. The command verifies every checksum before touching anything, refuses while a hub holds the store, and refuses a non-empty data directory unless you pass --force. It stages beside the destination and swaps by rename.

You should now see

agent-hub check prints checked N engine files, no problems and exits 0. On a restore, the command prints restored N files, M bytes, to <data directory>.

See also

• Upgrade and roll back - the pre-migration store a rollback uses
• Monitoring - offline check, doctor, and the integrity sample
• Configuration - backup_dir